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ABSTRACT 


The thesis provides lessons learned from experiments conducted by the 
11th Air Force to verify the capabilities of two vendor-produced Radar Data 
Management Systems (RDMS). The first part of the thesis provides background 
information explaining the impetus for such experiments and why a lessons 
learned approach was taken. The experimental plan and the final report from the 
PACAF experiments are analyzed using evaluation tools taught in the C3 
curriculum at the Naval Postgraduate School. The lessons learned from the 
mistakes made during these experiments are applied to produce a revised 
Experimental Plan. A lessons learned section follows the analysis. This section 
discusses specific lessons learned from the 11th Air Force experiments as well as 
more general lessons learned by the author. The thesis concludes with two 
chapters that provide overall conclusions and a summary, and recommendations 


for future work that can be accomplished in the area of radar data management. 
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I. INTRODUCTION 


Pacific Air Forces (PACAF) is presently acquiring a radar data management 
system (RDMS). This system is intended to both reduce the amount of radar 
clutter presented to the operator and reestablish radar sensitivity which was 
deliberately reduced for reasons to be given below. The underlying problems 
that would be solved by acquiring a RDMS were due to the fact that radar 
technology was upgraded faster than the technology of the computers that 
process the radar data in the air defense command and control (C2) architecture. 
In 1983, a new generation of radar became operational. The gap between radar 
and computer technologies immediately created a problem: the collective input of 
radar data overwhelmed the central computer. To relieve this burden, engineering 
change proposals (ECPs) were implemented to reduce radar sensitivity and, thus, 
the amount of data forwarded to the central computer. 

The ECPs were intended as a temporary solution, but they are still being 
used. The ECPs arbitrarily filter radar data and, as a result, actual targets may be 
filtered out, and air defense operations may be compromised. A more permanent 
solution is needed. 

In 1990-1991, two defense contractors claimed to have developed radar data 
management devices that exploited new computer technology to reduced the 
volume of data in a more selective way. By using these devices, the radar could 
be restored to full operational capability, and the volume of data could still be 
kept small enough for the existing C2 computers to process. The possibility of 
overloading the computer would be small. 

PACAF is anxious to exploit this technological improvement. It would allow 


them to better fulfill their mission requirements and still delay procurement of a 


new generation of air defense computers. Before committing to this course of 
action, PACAF wanted to assess the validity of the two manufacture's claims, and 
identify the best system. 

An experiment was conducted by the 11t) Air Control Wing (ACW) at 
Elmendorf AFB, Alaska. The data collection had to be done in a manner that 
would not interfere with the performance of the Region Operations Control 
Center (ROCC) operational mission, but actual field trials were preferred to 
laboratory tests. As a result, the data collection operation was "piggybacked" 
onto normal operations. 

When the 11} ACW attempted to analyze this data to see whether the 
manufacture's claims were true and to determine which manufacture's equipment 
performed better, they could not. They found that the massive volume of data 
collected did not include the information necessary to fully answer the questions: 
"Will the manufacture’s equipment eliminate data on tracks necessary for the 
conduct of air operations? " The 11th ACW needed to be able to identify the 
tracks, but that information was not available. 

The purpose of this thesis is to critically examine this case and document 


what went wrong. General questions such as: 


can experiments be successfully conducted within the operational 
environment? 


will be addressed, but the pnmary emphasis will be on the PACAF experiment. 


Il. BACKGROUND 


This chapter provides background information that is necessary to 
understand the following analysis of the PACAF experiment. The Impetus For 
Change Section explains the radar data management problems and why a 
solution is needed. The System Description Section describes the components 
directly involved with radar data management. A description of the RDMS as a 
potential solution along with the anticipated benefits to the command and control 
system are addressed in the Radar Data Management Equipment Description 
Section. Finally, the Special Considerations Section illustrates the challenges of 


conducting an experiment in an operational environment. 


A. IMPETUS FOR CHANGE 

The Alaska NORAD Region (ANR) was driven by several factors to change 
the way radar data was managed. The decline of the Soviet threat and resulting 
draw down of the American military machine was a principal factor. In 1990, the 
U.S. Air Force began a slow process of reorganization, within which the Alaskan 
Air Command (AAC) ceased to be a major command. It became the 11th Air 
Force, under the organizational command of Pacific Air Forces (PACAF). The 
reorganization meant that the 11th AF now had to go through PACAF for all of 
its funding, including justification of the funding for existing programs. 

The eruption of Mount Pinatubo in Philippines and resulting loss of Clark Air 
Base was another factor. Clark Air Base was the location of an air training 
exercise called Cope Thunder, a major exercise PACAF used to train its forces in 
Asia. PACAF needed to fill the void left by the loss of Clark AB and thus Cope 
Thunder, and Alaska was seen as the best replacement. The Alaskan theater 


provided many opportunities: large training areas over sparsely populated areas; 


the capability to add more training areas; in-theater tanker support; two large 
bases capable of handling an expanding number of fighter squadrons as well as 
accommodating visiting squadrons; a well established command and control 
network (ANR ROCC and AWACS (Airborne Warning and Control System)); 
and new developments for integrating theater wide command and control 
facilities that were already in progress (i1.e., Commander's Theater Information 
System (CTIS) and Aircraft Combat Maneuvering Instrumentation (ACM))). 

PACAF decided to move Cope Thunder to Alaska, and plans were put 
together to expand the training ranges. But, radar coverage within the proposed 
training areas was not adequate to provide safe operations. Additional radars 
were needed, and an initial increment of two was requested to increase low level 
coverage. To be effective, the additional radars had to be integrated into the 
ROCC, but the central computer is a weak link because of its limited processing 
capability. Therefore, adding more radars, especially radars concerned with low 
level coverage, which produce more clutter than radars concemed with high level 
coverage, meant a much greater chance of overloading the ROCC central 
computer. 

Radar data management became a key issue. Given the factors of limited time 
(PACAF was proceeding rapidly with the range improvement project), limited 
funding (most of the funding was going for range improvements), limited 
computer processing power of the ROCC computer, and limited sensitivity of 
existing radars caused by the ECPs; the 11th ACW staff knew one possible 
answer. They had been working on the issue of radar data management for over 
a year and knew of commercial-off-the shelf equipment that would be low cost, 


quickly installed, reliable, maintainable, and interoperable with the existing and 


new radars and with the ROCC. This new equipment was available from two 
defense contractors. 

The 11th ACW was tasked with investigating both potential solutions and 
making recommendations about which, if any, system to use, and how to use it. 
The first major task was to confirm the claims made by the defense contractors. 
Experimental plans were drawn and experiments were run to test the systems. 
Would the RDMSs allow the ECPs to be removed from the radars to restore their 
sensitivity, and at the same time reduce the amount of clutter being sent to the 
central computer without eliminating valid air tracks? 

The 11th ACW staff was in charge of planning and executing the experiment, 
analyzing the data, and making recommendations about the radar data 
management systems. Shortly after the experiments concluded, the author, who 
was in charge of the experiment, was accepted to the Naval Postgraduate School 
(NPS). The data had not been closely analyzed, and with the training received at 
NPS, a thorough analysis finally could be done. Looking back, many mistakes 
had been made with the experiment, so many that a comprehensive analysis could 
not be done. Therefore, this thesis will examine what went wrong, propose a new 


plan, and provide lessons learned. 


B. SYSTEM DESCRIPTION 

The RDMS is aimed at solving the radar data management problem of the 
ANR, specifically the ROCC. The ROCC is the center for command and control 
in Alaska. This command and control architecture is described in the following 
paragraphs. First, roles and missions are examined, then the system components 


and their roles in radar data management are addressed. 


1. Roles and Missions 
Command and control is unique in Alaska, literally serving two masters. 
Operationally, the ANR reports directly to NORAD Headquarters. The mission is 


to provide "...warning of a surprise attack against North America, and 
surveillance, control, and defense of our sovereign airspace [Ref. 1]}." 
Administratively, the 11th Air Force reports directly to PACAF for everything it 
needs to carry out its assigned mission. 
2. System Components 

Two components of the ROCC are directly involved with the radar data 
management problems, the long range radars, better known as Minimally 
Attended Radars (MAR) and the ROCC central computer located at the ROCC. 
They are described below. The radar data flows from the MAR to the ROCC 
central computer via a commercially leased satellite. The flow of data is depicted 
in Figure 1 [Ref. 2]. 

a. Long Range Radars 

Long range radars are the heart of the Alaskan air defense network. 

Figure 2 depicts the approximate radar locations in Alaska. In 1983, a new type 
of radar became operational. "The AN/FPS-117 Minimally Attended Radar 
(MAR) is designed to provide long-range accurate aircraft identification and 
position data..." The MAR provides "...digital output messages containing range, 


azimuth, and height information for radar and beacon targets.” [Ref. 3]. 
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Two radar-related factors contribute to the radar data management 
problems. The first factor is the extreme sensitivity of the radar. The unclassified 
operational characteristics of the radar provide an insight. Rotating 360 degrees 
every 12 seconds, the radar search volume is from 5 to 200NM, +6 to +20 
degrees elevation, a 100,000 feet altitude limit, range resolution of .SNM, and a 
range accuracy of .25NM [Ref. 3]. 

The second eOnibutine factor 1s the adverse clutter environment 
found in Alaska. "The clutter environment in Alaska is difficult for radar 
operation because of unique terrain, a wide range of temperatures, extreme 
weather, and a large migratory bird population [Ref. 3]. 

b. ROCC Central Computer 

The ROCC Central Computer is the central point for processing the 
data received from all of the radar sites. This fully redundant computer is the 
Achilles heel of the entire data management process. (Figure 3 provides a block 
diagram of the fully redundant ROCC Central Computer [Ref. 4].) The main 
problem lies in the centralized role the computer plays. In every 12 second 
Sweep, each radar forwards all of the data detected within its search volume. The 
data remains unprocessed until it reaches the computer at the ROCC. With 18 
MARs located in Alaska, a vast amount of data must be processed by the 
computer. The computer processing time is synchronized with the radar sweep 
time of 12 seconds. If the computer is overloaded by too much data, then it 
arbitrarily "dumps" data in order to meet the 12 second processing time. Avoiding 
overloading the computer was the driving force for the clutter-reducing ECPs 


originally installed at the radars. 
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A conflicting sequence of events has occurred: the new radars 
produce too much data because of their sensitivity and the location of each 
within a high clutter environment; the data from all 18 radars must be processed 
by older, slower computers; and the clutter reducing ECPs that were enacted to 


avoid computer overload sacnifice the sensitivity that the new radars provide. 


C. RADAR DATA MANAGEMENT EQUIPMENT DESCRIPTION 

Two vendors presented potential solutions to Alaska's radar data 
management problems. Litton Data Systems Advanced Tracking System (ATS) 
and Sensis Corporations Multi-Scan Correlator Model 2000 (MSC-2000). Both 
of these devices work on similar principles, based upon software algorithms 
designed to discriminate tracks from clutter. 

1. General Description 

Each radar data management device 1s a computer, with the hardware 

and software necessary to reduce the amount of clutter caused by environmental 
factors, while at same time maintaining a high level of radar detection. The heart 
of each device is the tracking algorithm. The specifics of these algorithms are 
considered proprietary to each vendor, and will not be addressed here. Generally 
speaking, however, the tracking function is performed by an adaptive algorithm. 
The algorithm is designed to detect and establish a track within a certain number 
of radar scans under varying amounts of search volume, store the track 
information in a file, and compare new track information to the existing file. The 
track information is kept in the file for a certain number of radar misses before the 
track is dropped. In order for the tracking algorithm to be effective, the radar's 
search volume is divided into thousands of azimuth and range cells. The tracking 


algorithm uses these cells to establish the validity of a track. Once a track is 


I] 


established the RDMS sends only the track data back to the ROCC Central 
Computer [Ref. 5 and 6]. 
2. Benefits 

One approach to solving the underlying problem in Alaska's radar data 
management is decentralized processing of the radar data. The Litton Data 
Systems and Sensis Corporation devices focus on this approach, decentralizing 
the task of radar data processing. If all 18 radars could preprocess their own raw 
data, forwarding only processed track data, it would significantly lessen the 
processing burden on the central computer, thus lessening the chances of 
overload. Several benefits can be immediately realized: 

a. Restoring sensitivity to the radars without fear of overloading the 
computer will provide a higher level of operational capability. In the past, low 
level radar coverage was limited because it was the source of most clutter. The 
ECPs primarily focused on reducing the low level radar beam coverage. But, low 
level coverage would not be a problem using the vendor's devices because tracks 
could be discriminated from the clutter, and only track data would be passed on 
to the command and control computer. Lowering the radar beams would benefit 
both missions of the ANR: air defense, including detection of air-to-surface 
missiles; and training, which requires maintaining separation of aircraft for the 
safe conduct of operations within the training range. 

b. The procurement of the next generation ROCC computer could be 
delayed. The limitations of the computer processing would be overcome not by 
developing a new ROCC computer, but by taking advantage of current computer 
technology to provide decentralized processing. A new ROCC computer would 
take many years to develop and cost more money than anyone would want to 


spend for an air defense command and control system. Research and 
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development for the vendor's RDMS is already accomplished--the systems 
already exist. The only task would be to ascertain the validity of the claims being 
made about these systems and determine which system best solves the radar data 
management problems. 

c. Integration and interpretability are additional benefits. By reducing 
the workload of the central computer, additional radars could be easily integrated 
into the existing air defense network. This is especially important because the 
range improvement plans call for adding two additional radars. The radars under 
consideration are Westinghouse radars developed for the Marines and used to 
provide short range, low level radar coverage. Both vendor's RDMS are 
interoperable with a wide range of radars. This will provide flexibility when 
acquiring additional radars for future expansion of radar coverage. Deployed 
radars equipped with an RDMS device would also be able to provide track data 


to the ROCC computer for display. 


D. SPECIAL CONSIDERATIONS 

Operational mission constraints and legal ramifications are necessary 
background. Special considerations, driven by the operational mission, had to be 
accounted for in the planning and execution of the experiment. Whether or not 
these factors had a direct effect on the outcome of the experiment is hard to say. 
However these factors are worthy of mention. They illustrate the challenges of 
conducting experiments in the operational environment. Legal ramifications were 
always a topic of utmost importance. The commander emphasized that he did not 
want to give the perception that the 11th AF was in the business of conducting 
its own test and evaluation. Additionally, he did not want to give the impression 


that he was bypassing the acquisition process. Therefore, he informed all 
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involved with this project to make it clear: 11th AF had not gone into business 
for itself. 
1. Operational mission constraints 

The experiments would have to be conducted on a non-interference 
basis. The operational missions could not be sacrificed at the expense of the 
experiments. This criterion was the ultimate requirement and provided the 
boundaries for the experiment. A direct cause and effect relationship can be seen 
between the operational mission constraints and variable selection in the 
experiment. Furthermore, controlling factors in an operational environment is 
much more difficult than in a laboratory. 

The 11th ACW staff wanted to vary certain factors, but was constrained 
by operational requirements. The factors that were affected were site selection, 
time, and type of radar. Only three sites were allowed as part of the experiment. 
Even these sites were subject to being removed if they were needed to fulfill an 
operational mission. The effect this had on the experiment was that it was now 
limited to the terrain, air traffic volume, and clutter unique to the three areas 
selected. This lead to several questions: Is this sample representative of the 
Alaskan environment as a whole? Could valid conclusions be drawn for the 
entire area of responsibility or just those sections? What about the areas where 
the new radars would be used? Since none of the radars selected is located in the 
proposed range improvement area, would the conclusions drawn from where the 
experiments were conducted be valid there? 

Time was also affected by operational constraints. The time of year was 
limited to the early winter season. Would this be a representative sample for the 
type of traffic volume and clutter seen throughout the entire year? Additionally, 


the time of day was limited. The experiment could only be conducted before or 
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after the daily training sorties were flown. This meant either early morning or late 
afternoon/early evening. Again, would this represent an environment similar to 
one normally seen? 

The radars used were a factor that should have been varied but could 
not be. Ideally, two different radars should have been examined: the MAR radar 
and the Marine short range radar. However, since the Marine radar had not been 
procured, only the MAR radar could be examined. 

Weather, traffic volume, terrain, and clutter environment are all factors 
that the experimenter would like to either control or vary but were literally at the 
mercy of the operational constraints. These factors are all functions of the site 
selection and time. Therefore, holding them constant or allowing them to vary 
was impossible and reflected the difficulty of conducting controlled experiments 
in an operational environment. 

2. Legal Ramifications 


, 


The term "demonstration plan" was used to emphasize the point that 
operational units were not in the business of test and evaluation. The 11th AF 
was conducting a demonstration of the equipment, at no cost or legal obligation 
to the government. Furthermore, the staff was going to examine the data the 
vendors collected to verify the claims of the vendors. The 11) AF did not want 
to give the perception that this unit was bypassing, or undermining, established 
acquisition procedures. In fact, the unit did not actively pursue vendors. The 
vendors made their own solicitation and offers of demonstration, at no cost to the 
government, and these offers were made prior to the range improvement project. 
Up until this point, Litton Data Systems was the only vendor known to 


have a radar data management device. Not until after completion of the Litton 


Demonstration was it learned that there was another vendor that had a similar 


is 


device and wanted the opportunity to demonstrate it in Alaska. This created a 
potential legal problem. If, Sensis Corporation, the other vendor, wanted to 
demonstrate their equipment and were turned down, then the government could 
face a lawsuit. In order to afford Sensis the same opportunity as Litton, it was 
decide that Sensis could demonstrate their equipment. Because of the short 
notice of the Sensis demonstration, another Demonstration Plan was not 
produced. Instead, the ATS Plan was followed and the results of both 
demonstrations were published in the Long Range Radar (LRR) Enhancements 


Staff Study. 


E. CONCLUSIONS 

This chapter provided background information and serves as a prelude to the 
analysis of the experimental plan and report. The Impetus for Change Section 
illustrated why the 11th AF felt management of the radar data was important. The 
System Description Section detailed the components involved with radar data 
management. The shortcomings as well as the potential benefits of installing an 
RDMS were elaborated in the Radar Data Management Equipment Section. 
Finally, the chapter ended with a discussion of the special considerations and 
their affect on the experiment. With the background firmly established, an 
analysis of the experiment can be conducted. The experimental plan and report 


will be analyzed in Chapter III. 
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lil. ANALYSIS 


This chapter presents an analysis of the Advanced Tracking System (ATS) 
Demonstration Plan and the LRR Enhancements Staff Study [Ref. 7 and 8]. From 
this point on, the ATS Demonstration Plan will be referred to as the Plan and the 
LRR Enhancements Staff Study will be referred to as the Report. The reader is 
encouraged to read both the Plan and the Report which can be found in 
Appendices B and C. This will aid in understanding the analysis. 

The Plan and Report analyses follow a simple format--mistakes are pointed 
out followed by potential solutions. An Experiment Plan and Reports Format and 
an Evaluation of Studies Paradigm that were presented in the C3 curriculum at 
the Naval Postgraduate School were used to analyze both the Plan and the 
Report [Ref. 9]. These tools are used to identify mistakes and to provide 
guidance for the solutions. The Experiment Plan and Reports Format and the 
Evaluation of Studies Paradigm are included as Appendices D and E. At the 
conclusion of the analysis, a newly revised Plan is presented that integrates all of 
the lessons learned in the Plan analysis (Appendix F). However, there is no newly 
revised Report. The main reason Is that the data necessary to provide a thorough 
analysis is not available; therefore, major sections of the report would be missing. 

The Experiment Plan and Reports Format serves as a model of how a plan 
and report could be organized and what elements should be contained in the 
Structures of plans and reports. From this point on, the Experiment Plan and 
Reports Format will be known as the model (Appendix D). 

The Evaluation of Studies Paradigm (Paradigm) is an essential ingredient in 


the analysis because it defines the elements of the model (Appendix E). Together 
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the model and the paradigm provide the framework needed to analyze the Plan 


and Report. 


A. PLAN ANALYSIS 

The ATS Demonstration Plan lacks a clear statement of the purpose of the 
experiment. Neither the real world problems that the ATS is intended to solve, 
nor how the results of the demonstration will be used to show that it can, are 
clearly stated. As a result, the following sections lack foundation. They should 
give details about the conduct of the experiment and the analysis of the results, 
and show how everything is specifically organized to answer the questions at 
hand. Without clear articulation of the problem, this is not possible. Detailed 
analysis of each section, as defined in the model, followed by a sample 
introductory revision is given below. 

1. Mistakes (Introduction) 

An experimental plan should be a cohesive document that lays the 
foundation for the entire experiment, up to and including the analysis. It should 
provide introductory material saying who is performing the experiment, why the 
experiment is being performed (purpose), and defining the experimental scope. 
This material should clearly identify the real world problem that the experiment 
will help solve and the specific questions the experiment seeks to answer. 

The major problem with the Plan is its lack of a unified introduction 
section containing clearly defined objectives of what the experiment is supposed 
to do. The introductory elements that the Plan does contain are scattered 
throughout the plan as opposed to being incorporated into a unified introduction 
section. For example: the team is introduced in section 2-1; the purpose 
Statement is found in section 1-1; and the objectives are found in section 4-1. This 


creates a lack of focus for the entire experiment. 
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The purpose is not fully addressed. There is a purpose statement, but it 
is rather weak. The Plan states the"...demonstration is to determine whether or 
not the ATS can be used in the Alaskan Theater to improve radar performance 
and make recommendations concerning the desirability of [incorporating] the 
ATS in the Alaskan Surveillance and Command and Control System." The 


1 


purpose also mentions that the primary objective is to "...collect sufficient data to 
determine the ATS's capability for substantially improving the probability of 
detecting targets (at lower altitudes and clutter environment).” But, the real 
world problem is neither poor radar performance nor low probability of detection. 
The actual real world problem is radar data overloading the central computer. 
This 1s not mentioned. Furthermore, there are no specific questions that the 
experiment seeks to answer. So naturally, the approach the experiment will use 
to answer the research questions is not addressed and neither are the anticipated 
results. The scope of the experiment is also not addressed. This section should 
explain why this experiment is important to the success of the mission, what 
constraints are imposed on the experiment, and what the impact of the 
experiment will be. 

The stated objectives are not appropriate because they are benefits 
rather than objectives, and they do not map back to the primary objective listed in 
the purpose statement: to provide evidence as to whether or not the ATS is 
capable of improving the probability of detecting low level targets in a high 


clutter environment. 
Objective one states: 
Reduce demands on the FYQ-93 ROCC central computer; processing time 
and memory space. At present, central computer processing time and memory 


storage space is nearly saturated; while memory storage in the processor 
controller is overloaded to the point it “dumps’" data. 
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Objective three states: 


Increase probability of detection by reducing human variability in the 
detection/tracking process. The ATS serving as a front end processor at the 
radar will automatically detect, establish, and pass target data to the Region 
Operations Control Center (ROCC). The presentation of only true targets in 
the ROCC will eliminate operator decisions concerning the validity of targets. 
This will increase the probability of establishing air tracks. Also because only 
true targets are presented, maintaining track continuity will improve for both 
computer tracking algorithms and when operators override computer tracking. 


Neither one of these are valid objectives. Both of them are benefits that 
would result if the ATS works. However, determining the truth of this assumption 
is the primary objective of the experiment. 

Objective two states: 


Radar Enhancement: Regain loss of probability of detection, sacrificed at 
the expense of clutter reduction solutions to the radar. With the ATS's ability 
to extract true targets from clutter, the radar beam could be lowered thus 
increasing the probability of detection of low level targets. 


This objective is not appropriate because it is the approach that the 
experimenters are taking to test the ATS's capabilities. In order to validate the 
manufacture's claims about the ATS, the radar must be allowed to operate in its 
fully operational state uninhibited by any ECPs. The purpose is determine the 
amount of data being processed by the radar with and without the ATS, and to 
determine whether or not the ATS is eliminating valid target data. 

Objective four states: 

Role of the ATS in Alaska: By analyzing the data collected during the 
demonstration, we will draw inferences concerning the utility of the ATS in 
Alaskan Theater. Areas of interest: advantages of incorporating the ATS at 


each AN/FPS-117 in Alaska from a sensor interoperability standpoint; added 
measure of safety in large scale exercises. 


This objective is neither a benefit nor an approach. Instead objective 
four is a conclusion that can only be made after the data is collected and the 


results analyzed. 
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2. Solutions (Introduction) 
The major problem with the Plan is its lack of a unified introduction 
section containing clearly defined objectives of what the Plan is supposed to do. 
The solution is to create an introduction section that corrects the mistakes. An 


example of what the introduction section should look like follows: 


I. REVISED INTRODUCTION 
A. Project Management 


The 11 ACW is going to conduct an experiment to assess the 
capabilities of the Litton Data Systems Advanced Tracking System 
(ATS). 


1. In order to manage the demonstration, an Advanced 
Tracking System Project Management Team was established. The 
demonstration team leader is 118 ACW/DOX; other team members 
are listed in paragraph three. 


2. The following agencies are participating or providing 
support for the ATS demonstration: 


a. 11 ACW/DOXXQ__ -- Capt Pierce 


b. 11 ACW/DOP -- SSgt Myher 

c. 11 ACW/LGOR -- MSegt DeLuca 
d. 11 ACW/LGKC -- MSet Shuler 
e. 11 ACW/LGKM -- TSgt Cushman 
f. 744 ADS/DOO -- Lt McNeil 

g. 21) StTFW/DOW ~ -- Capt Hill 
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B. Purpose 


The purpose of this experiment will be to determine whether 
or not Litton Data Systems Advanced Tracking System (ATS) is 
capable of more efficiently managing the flow of radar data coming 
into the Region Operations Control Center (ROCC). Managing the 
radar data more efficiently will enable the operators to realize the 
full operational capabilities of the new radars without overloading 
the ROCC Central Computer. As a result, the demonstration is 
designed to provide insight into the radar data management 
problems being experienced by the Alaska NORAD Region (ANR) 
and the ability of the ATS to solve them. Specifically, the ANR 
would like to determine the validity of the claims of the ATS's 
capabilities. For example: Can the ATS accurately discriminate true 
targets from clutter? From an air defense point of view, can the ATS 
provide target discrimination without target elimination? 


The demonstration will select three sites, all equipped with 
ECPs. All of the ECPs will be turned off for the demonstration in 
order to reestablish the radars to a fully sensitized state. The ATS 
must prove that it is capable of handling the data workload from a 
fully sensitized radar. 


The ATS will not be installed at the selected radars because 
of the logistics and prohibitive costs. Instead, the ATS equipment 
will be installed in the ROCC. 


The ATS will be connected at the ROCC Communications 
Segment, the point where the data arrives in the ROCC via satellite 
from the selected radars. The data arriving at this point has not yet 
been processed by the ROCC central computer. Therefore, 
connecting the ATS at the ROCC Communications Segment 
simulates the ATS processing the data at the radar site. 


The next step will be to compare the amount of data a fully 
sensitized radar forwards with the amount of data a fully sensitized 
radar equipped with an ATS forwards. The ROCC provides the 
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capability to simultaneously measure ATS and non-ATS data from 
the same site (see Figure 4)[Ref. 10]. The reduction in data, and the 
quality of the data must be analyzed simultaneously. The ATS may 
live up to its clutter reduction claims, but the data has to be 
analyzed to ensure that the ATS is not eliminating or erroneously 
manipulating valid air targets necessary for air defense and air 
training missions. In order to verify the target data, a specific 
number of aircraft, flying different profiles, will be used. The 
experiment will be designed to control the number of aircraft. 
However, the actual flight profiles along with the number of aircraft 
used during the test will only be known by a third party, 
disassociated from the experiment. Only after the experiment is 
completed and the analysis turned in will the aircraft portion of the 
experiment be revealed. The purpose of this single blind design is 
to eliminate any bias from the experimenters. 


The anticipated results: Measurable differences between the 
quantity of data received from a radar in a fully sensitized state, 
when equipped with an ATS and when not, are expected. 
However, whether or not the ATS eliminates valid air targets 
remains to be seen. 
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C. The scope of the demonstration: 


This demonstration will provide the information the 11th AF 
Commander needs to decide whether or not the ATS (or other 
associated Radar Data Management System) provides viable 
solutions to the radar data management problems experienced by 
the ANR. There are several potential benefits if the ATS 
demonstration proves successful. First, ECPs can be eliminated from 
the radars, restoring them to their full operational capability, and 
providing greater low level detection of aircraft and missiles. In the 
area of training, the improved detection capability will increase the 
radar coverage in the low level training areas. 


Benefits should also be seen in the ROCC central computer. 
Installing the ATS at the radar head will decentralize the computer 
processing. The ATS will discriminate between clutter and target 
data, dump the clutter data, and pass only the target data to the 
central computer for further processing. The main benefits to the 
central computer are reductions in the amount of processing 
required and the potential for a computer overload that arbitrarily 
dumps data. 


The reduced computer workload will enable the installation 
of additional radars. This is important because PACAF wants to 
expand the air training ranges in Alaska. The eruption of Mount 
Pinatubo and subsequent loss of a major air training exercise in the 
Philippines called Cope Thunder required more training, and thus an 
expansion of the air training ranges, in Alaska. As a result, 
additional radars are needed to provide coverage in these new 
training areas. However, the additional data from these sites could 
easily create an overload condition in the computer. Dumping data 
during a major exercise would seriously jeopardize the safety of the 
entire mission. 


The life of the central computer will be prolonged because 
the amount of data the central computer has to process will be 
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eased. Thus, the costly acquisition of a new generation ROCC 
computer can be delayed. 


3. Mistakes (Experimental Design) 

The details of the experimental design should be included in the plan. It 
should explain the experimental setup in detail, including the physical setup, who 
the test subjects or experimental units are, what special equipment is required, and 
give the trial schedule. The hypotheses that are to be examined should be stated 
specifically, as should any assumptions that will be made. The statistical design of 
the experiment should be explained, along with definitions of measures that will 
be employed and instrumentation that will be used to collect the data. Provisions 
should be made for testing the setup and design, and for the trials. 

While the Introduction section of the Plan fails to provide the necessary 
focus for the experiment, the Experimental Design section suffers from a lack of 
detail. The closest the Plan comes to an experimental design is the Procedures 
Section, a subsection of the Demonstration Objectives. 

The Procedures Section attempts to cover too many details in a single 
page of material. Of the five statements listed on page 4-2, only three can be 
considered procedures. These three statements, taken from the Plan, are quoted 
below: 


Three radar sites (Cape Romanzof, Kotzebue, and Tatalina) will be used to 
collect data to satisfy objectives one and two (reduce demands on the FYQ- 
93 ROCC central computer, and radar enhancement). These sites were 
chosen because they possess the latest clutter reducing software (i.e. ECP 
175 and the Multiscan Detection Processing (MDP)). At different times each 
day, data will be collected from each site for a period of 105 minutes. Each 
fifteen minutes, within the data collection window, the radar will be placed 
into a different configuration (e.g. ECP 175 enabled and MDP disabled). Data 
will be collected via data reduction, later collated on the Site Data Sheets 
(figure 1) and analyzed. 
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Data will be collected using low level flights (if available) in the ACMI 
range. This data will be used to assess the ATS'ss ability to enhance the 
probability of detection (Pd) of the FPS -117 radar. 


A tracking test will be used to meet the third demonstration objective 
(increase probability of detection by reducing human variability in the 
detection/tracking process). A operator will be selected at random and given 
an area of responsibility (AOR) to initiate and maintain track continuity on all 
tracks in the AOR. The operator will operate with and without the benefits of 
the ATS and all track histories will be recorded via data reduction. A 
comparison will be made of the data collected from the data reduction 
products to determine whether or not an operator will realize an increased 
probability of detection by reducing human variability in the 
detection/tracking process. Detailed instructions will be provided to each 
surveillance operator by the project coordinator--thus insuring a degree of 
procedural standardization. 


These three statements seem to try to map back to the original objectives 
that were not clearly defined in the first place. But, the result is unsuccessful, and 
the reader is left confused as to the specific outcome the procedures are trying to 
accomplish. The lack of clearly defined objectives has an adverse effect on how 
the experiment is to be conducted. As a result, the reader comes away without a 
clear idea as to what problem the experiment is trying to solve. 

Another problem is that there are no procedures describing how the 
objective of "Reducing the demands on the FYQ-93 ROCC central computer..." 
will be accomplished. Is this objective no longer important? 

The arbitrary presentation of procedural details further illustrates a lack 
of focus in the Plan. Paragraph two states that three sites were selected based 
upon the installation of certain ECPs. Why does it matter which ECP is installed 
at each radar? Should not one of the main procedures of the experiment be to 
disable all of the clutter reducing ECPs? If this is the case, then it should not 
matter what kind of ECP is installed at the radar. The procedure suggests that the 
experiment is designed to compare the various ECPs located at different radar 


sites with the ATS. This procedure does not map back to any of the objectives. 
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To further illustrate the arbitrary nature of the procedures, paragraph 
two States that data will be collected for a period of 105 minutes, and that every 
15 minutes the radar will change to a different ECP configuration. There is no 
explanation given why a 105 minute data collection period was chosen. 
Furthermore, manipulating the configuration of the ECPs is unnecessary. The 
details mentioned in paragraph two only distract and confuse the reader. 

Finally, paragraph two mentions that data will be collected via data 
reduction and then analyzed. Several questions are not addressed. First, what 
kind of data is being collected? How will the researcher know if this data will 
provide answers to the research questions if it is not clear what the questions are? 
What will the data measure? How will the data be collected? The data will be 
collected by data reduction, what does that mean? What instrumentation will be 
required to collect this data? What is the plan for analyzing the data after it is 
collected? All of these questions, and more, need to be included as details in the 
design of the experiment. 

Paragraph three states: "Data will be collected using low level flights (if 
available) in the ACMI range. This data will be used to assess the [ATS's] ability 
to enhance the probability of detection (Pd) of the FPS-117 radar." This is what 
the experiment is suppose to be about, but, no details are given as to how this will 
be carried out. The lack of detail makes the reader wonder whether or not the 
experiment will be conducted "on the fly.” 

Paragraph four is a procedure for another experiment that would 
determine whether or not the ATS will increase the probability of detection by 
reducing the human variability in the detection/tracking process. This procedure 
fails to define details of how the experiment will be executed. For example: the 


procedure states: "A[n] operator will be selected at random and given an area of 
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responsibility (AOR) to initiate and maintain track continuity on all tracks in the 
AOR.” What will his performance be compared to? What kind of operator will be 
selected? Background information is required here because there are different 
types and capabilities of ROCC operators. Furthermore, how will the operators 
be selected; based upon what criteria? Again, the issue of what measures will be 
used are not addressed. How will the data be collected? What instrumentation 
will be used to collect the data? What is the plan for analyzing the data? How 
does this experiment provide the evidence necessary to determine the capabilities 
of the ATS? Does this experiment focus on the primary objective? Or, is this 
experiment an example of how an experiment can get off track because of a lack 
of clear objectives? 
4. Solutions (Experimental Design) 

The experimental design problems are solved by utilizing the concepts 
found in the model and the paradigm. According to the paradigm, ‘the 
Experimental Design is comprehensive but executable if it tests all the pertinent 
hypotheses with measurable data by a practical analysis plan." In order to 
accomplish this task, the model must be used. The revised Experimental Design 
section that follows applies both the model and the paradigm and is a direct 


extension of the revised Introduction section. 
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IH. REVISED EXPERIMENTAL DESIGN 
A. Setup: 
1. Physical 


The physical setup is the crux of the entire experiment. The 
demonstration would not be possible if not for the flexibility of the 
ROCC system. Figure 4 depicts the actual physical interface 
between the ROCC and the ATS equipment. The following is a 
discussion of the ROCC-ATS interface. 


The digital switch provides the interconnection between the 
communication channels from the radar sites and the central 
computer. The radar data at the digital switch has not been 
processed by the ROCC computer. Each radar sends its data to the 
digital switch at the ROCC via three 2.4 kbps communications 
channels. These channels feed directly into three ports associated 
with each radar at the digital switch. Two ports, designated A and 
B, are normally interconnected with the primary and standby central 
computers. The third port, designated C, is available for recording 
data on a noninterference basis. This is the location of the ROCC- 
ATS interface. At the same time, the cable from port B of the Digital 
Switch to the central computer will be disconnected at the 
computer, and the output from the ATS will be connected to the 
standby computer at its input channel (Figure 4). 


2. Factors: 


There are three experimental factors: radar sites at three 
levels; Litton ATS at two levels; and analysis equipment at two 
levels. Three radar sites have been chosen: Kotzebue, Cape 
Romanzof, and Tatalina (Figure 2). The ATS radar processing 
hardware/software is either on or off. Finally, the equipment being 
used to analyze the data is either third party, Air Force approved, 
test and evaluation equipment or the Litton Data Systems analysis 
program associated with the ATS. 
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3. Special Equipment: 


a. The ATS consists of all of the interface hardware as well as 
the software to conduct the data collection and analysis. All will be 
provided by Litton Data Systems. 


b. A neutral third party will be involved to provide balance 
and legitimacy to the experiment. This third party (i.e., 84th Radar 
Evaluation Squadron) must conduct the experiment in parallel with 
Litton. They must interface at the same location, with the same site, 
and at the same time. They will collect the data, analyze it, and 
compare their results with Litton’s experimental results. 


Aircraft will be dedicated to fly certain profiles within the 
coverage limits of the selected radar. 


4. Schedule of Trials: 


The schedule of trials will not be formally addressed in this 
sample plan. The driving factor for the schedule of trials will be the 
availability of the aircraft needed to fly the flight profiles. Making 
up a schedule of trials does not provide sufficient lessons learned. 


B. Hypotheses: 


la. There is no difference between the amount of data 
forwarded by the fully sensitized radar and the radar equipped with 
the ATS. 


lb. The alternative: There is a difference between the 
amount of data forwarded by the fully sensitized radar and the radar 
equipped with the ATS. 


2a. There is no difference between the number of targets 
reported by the ROCC computer combined with the fully sensitized 
radar and the radar equipped with an ATS. 
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2b. There is a difference between the number of targets 
reported by the ROCC computer combined with the fully sensitized 
radar and the radar equipped with an ATS. 


3a. There is no difference between the third party analysis 
results and the Litton Data Systems analysis results. 


3b. There is a difference between the third party analysis 
results and the Litton Data Systems analysis results. 


The first two hypotheses deal with the issues of quantity and 
quality of the data the ATS produces. Does the ATS make a 
difference in the amount of data being sent back to the ROCC 
central computer? And, does the ATS discriminate targets from 
clutter without eliminating valid air targets? 


The third hypothesis is designed to validate the Litton 
demonstration by determining whether or not the data is consistent 
with that from an established test and evaluation system. The 11th 
ACW staff must be sure that the ATS works as claimed and that the 
analysis tools that are part of the equipment package portray the 
data in an accurate manner. 


C. Assumptions: 


1. Litton's interface hardware is compatible with the ROCC 
equipment. 


2. The software will provide the type of analysis needed to 
answer the research questions. 


3. The weather patterns and clutter density at the three sites 
are representative of the (AOR) overall. 


4. Both Litton and the third party Air Force evaluation 
equipment is capable of counting and keeping track of the number 
of targets received from the radar. 
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Note: This assumption is based on the claims of Litton that its 
software has the capability to compile this type of information [Ref. 
3]. Additionally, since the ROCC computer, a 1970s version 
computer, has the capability to count the number of tracks in the 
system, and 1s certified, it is assumed that the evaluation equipment 
exists within the Air Force, or else the ROCC computer could not 
have been certified in the first place. 


D. Statistical Design of Experiment 


The experiment is a 3 X 2 X 2 factorial single blind 
experiment. This means that one factor has three levels and the 
other two factors each have two levels. The first factor has three 
levels because of the selection of three different radar sites. The 
radar sites associated with these levels are: Kotzebue, Cape 
Romanzof, and Tatalina. This factor corresponds to hypotheses one 
and two stated above. The processing equipment is the second 
factor. The two levels are whether or not the radar is utilizing the 
ATS. This factor also corresponds to hypotheses one and two. The 
third factor is analysis equipment. The two levels for this factor are 
the third party operationally approved equipment and the Litton 
analysis equipment. This factor corresponds to hypothesis number 
three. The purpose of designing a single blind experiment is to 
eliminate experimenter bias and to serve as a means to evaluate 
whether or not the ATS is eliminating valid air targets. 


E. Measures 


The experiment is designed to evaluate the data forwarded 
by the radar. The data will be collected and reduced into two 
categonies. The first category deals with the quantity of data and 
the second deals with the quality of data (target elimination). The 
categories of data are listed below. 


Category one: Quantity. The total amount of data processed 
and sent by the radar to the ROCC, measured in bits per second. 
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Category two: Quality. The total number of targets being 
sent by the radar to the ROCC, measured by the total number of 
targets being sent by the ROCC computer to the operator. 


F. Instrumentation 


The instrumentation is the equipment used by Litton and 
the third party Air Force evaluation team to collect and analyze the 
data. 


G. Testing & Pilot Trials 


Testing will be done to ensure the ATS and third party 
equipment can be connected to the ROCC digital switch. 
Additional testing will be done to ensure the commander that the 
experiment will not interfere with the daily operations of the ROCC. 


After the interfaces are made, pilot trials will be run to ensure 
that the data being collected is the correct data to test the 
hypotheses. The researchers will use the pilot testing to practice the 
experimental procedures. For planning purposes, time should be 
allotted in the master schedule for the pilot testing. 


5. Mistakes (Data Description) 

A description of the data should also be included in the plan. This 
section of the plan should contain an example of the raw data, (the entire set of 
collected data should appear in an appendix) problems experienced with 
collecting the data should be addressed, and the data should be presented in an 
understandable and easy to read format (i.e., tables or spreadsheets). Any 
methods used to reduce the data, or other manipulations, should be explained. 

The way the Plan describes the data is another contributing factor to its 
overall weakness. The Plan describes the data in a subsection entitled Data 


Products. The opening statement: “Data products required to ensure accurate 


34 


analysis.” implies that the data collected will answer the research questions being 
asked. The analysis of the Experimental Design section revealed that the plan 
does not mention how the data will be collected. The lack of measures, coupled 
with the fact that no testing and pilot trials are called for in the plan, leads one to 
believe that no one would know if the wrong data had been collected until after 
the analysis had begun. This is too late to learn that the data collected will not 
answer the research questions. 

Another problem with the Data Products section is that it states the data 
is to be collected from the Aircraft Combat Maneuvering and Instrumentation 
Center computer (ACMI) and the ROCC. However, there is no description of the 
data that will be collected from these two points. The Plan also mentions that a 
log will be maintained at the Air Surveillance Technician position. A log may be 
useful to help reconstruct the experiment, but it should be kept by a person 
directly involved with the experiment. Furthermore, a log of this nature does not 
provide any real data for analysis. 

6. Solution (Data Description) 

As with the other sections of the Plan, the revised Data Description 

Section that follows provides a possible solution based upon the lessons learned 


from the mistakes of the original plan and utilizing the model and paradigm. 


Il. REVISED DATA DESCRIPTION 


Table 1 presents the data coding scheme. The table depicts 
the three factors (Radar Sites, Analysis Equipment, Processing 
Equipment) along with their associated levels. The number in 
parenthesis 1s the code assigned to the particular level. The coding 
scheme will become apparent when the table listing the different 
combinations of trials 1s presented below. 


35 


TABLE 1 DATA CODING SCHEME 


Radar Sites Analysis Equipment Processing Equipment 
Kotzebue (1) Third Party (1) Non ATS (1) 


Cape Romanzoff (2) Litton (2) ATS (2) 
Tatalina (3) 





The data is divided into two categories. 


Data: Total amount of data received from the radar (X in bits 
per second) 


Total number of tracks (Y) 


The data is divided into two categories to aid in the analysis, 
but it will all be collected at once for each trial. The computer 
software of the evaluation equipment will reduce the data into the 
X and Y categories designated above. 


Table 2 depicts the combinations of levels of the factors. 
There are twelve combinations of factor levels, found by multiplying 
one factor with three levels and two factors each with two levels. 
Each of the trials will be replicated thirty times in order to assume 
the data approximates a normal distribution. 
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TABLE 2 TRIALS 


Trial # Radar Sites Analysis Equip. Processing Equip. Data (X) Data (Y) 


Oo oo ~) OM BW HN — 
bo WN =| — PN = — NON — = 
Nom NOS NO — NO — NK NH — 


1 
1 
l 
J 
2 
2 
2 
2 
5 
3 
5 
5 





There are a Several steps in the data collection process. The first 
step is to collect all of the data the radar is sending to the ROCC. 
The total amount of data will be used by the analysis to answer 
hypotheses one and three. Target counts will then be compiled 
using the software capabilities of the computers. The target counts 
will be used to answer hypothesis number two. Additionally, the 
target count data can be used to provide supplementary information 
concerning hypotheses numbers three. 


7. Mistakes (Analysis Plan) 

A good plan must state how the analysis will be performed (the details 
of the analysis belong in the report). The analysis plan must also state the 
Statistical tests that will be used to analyze the data. 

The Plan does not address how the data will be analyzed, only that once 


the data is collected it will be analyzed. 


oy, 


8. Solutions (Analysis Plan) 
The analysis section is the final section of the revised plan. Like the 
sections before, the mistakes made in the original plan are overcome by using the 


model and the paradigm as a guide. 


IV. REVISED ANALYSIS PLAN 


Balanced Analysis of Variance (ANOVA) will be used to 
analyze the data. The results will determine the outcome of the 
hypotheses tests. As indicated in the data section, there are two 
parts to the plan. The first section will use the balanced ANOVA to 
analyze the total amount of data. Thirty replications of each of the 
twelve trials will be made. This will be done to provide normality. 
The second phase of the analysis will also use balanced ANOVA, 
this time to analyze the thirty replications of target counts. 


B. REPORT ANALYSIS 

The same analytical tools used to evaluate the ATS Plan will be used to 
analyze the Report. The analysis will compare the information in the Report to 
what should be included in a report based upon the model and the paradigm. As 
with the analysis of the Plan, mistakes are addressed and solutions are presented. 
However, a revised Report is not presented because data crucial to the analysis 
was not collected or is not available for analysis. Since the entire report revolves 
around the Analysis Section, and data is not available to be analyzed, merely 
restating what has already been said in the revised Plan and leaving the Analysis 
Section out would have little value. 

The formal title of the original report is: LRR ENHANCEMENTS STAFF 
STUDY. This is the final report to the 11th Air Force Commander recommending 


a course of action based upon the outcome of the RDMS demonstrations. It 1s 
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interesting to note that only one demonstration plan but two demonstrations are 
addressed in the final report. 

Two demonstrations were conducted by the 11 ACW. First, Litton Data 
Systems demonstrated the Advanced Tracking System (ATS). The planning for 
this demonstration took an entire year and culminated in the execution of the 
ATS Demonstration Plan. After the ATS demonstration and analysis were 
complete, Sensis Corporation hearing that their competitor Litton Data Systems 
had conducted a demonstration in Alaska, contacted the 11 ACW expressing 
their desire to demonstrate their equipment. The 11 ACW, concerned with the 
legal consequences of not letting Sensis Corporation demonstrate their 
equipment, allowed a Sensis demonstration. Because of the short time interval 
between the Litton and Sensis demonstrations, a plan could not be written, 
approved, and disseminated. As result, a Sensis Plan does not exist. The Report is 
the first time the Sensis demonstration is mentioned. 

1. Mistakes (Introduction) 

An Introduction Section of an experimental report is similar to that 
already discussed for an experimental plan. This section should introduce the 
researchers involved with the experiment, contain a well defined purpose, 
including a statement of the real world problem and specific questions the 
experiment seeks to answer, the experimental approach, and anticipated results, as 
well as the scope of the experiment. In addition, the Introduction Section of the 
report should also contain any additional information that may have occurred 
during the course of the experment. 

The Report fails the same the way the Plan fails. It does not meet the 
requirements of either the model or the paradigm. The Report does not contain a 


clearly defined Introduction Section. The analysis of the Plan demonstrated that 


oie, 


if the Introduction Section failed to state a clear objective the rest of the Plan 
suffered because it 1s too easy to get side tracked from the original purpose of the 
experiment. 

There is one overarching problem with the Report. There is a shift in the 
emphasis from what the Plan set out to accomplish and what the Report is trying 
to accomplish. The first sentence in the Report reveals the discontinuity between 
the Plan and the Report. The first sentence states: "The 11th AF needs to acquire 
a system that will integrate existing systems 1.e., 3DS [Digital Data Display 
System], AICU [Advanced Interface Contro] Unit], and ACMI [Air Combat 
Maneuvering and Instrumentation], into a cohesive, efficient system providing 
unseen benefits to the Alaska 2000+ Range Improvements Project" (see Glossary, 
Appendix A, for definition of terms). Remember, the Plan was to focus on 
validating the claims of Litton Data Systems Advanced Tracking System 
capabilities. However, the first statement in the Report clearly indicates a shift to 
a different emphasis. 

The next area the Report addresses is: Factors Bearing on the Problem. 
The first factor (paragraph a) states the requirement for integrating another type 
of radar into the ROCC. This is valid requirement. However, isn't this 
requirement a benefit? If the Plan and the execution of the experiment had 
focused on validating the ATS's capabilities, and the report presented evidence to 
support these capabilities, then the RDMS could be considered a viable solution. 
The next three factors, also taken from this section, address integration issues 


(paragraph b,c,and d) and appear below. 


b. Currently, no capability exists to automatically insert flight 
data on military aircraft into 3DS. The NORAD Software Support 
Facility (NSSF) is working on CSCP [Computer Software Change 
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Proposal] # 4569 which would allow the automatic tell of military 
aircraft. Hence, creating the pathway needed to get data on military 
aircraft into 3DS. The solution the NSSF is trying to achieve is a 
major software change: no solution is on the horizon. 


c. Track data on aircraft not equipped with ACMI pods, but 
still players in large force exercises (LFEs), are not entered into the 
ACMI. 


d. Radar data on aircraft not equipped with ACMI pods are 
detected by LRRs surrounding the range; however, this information 
is only sent to the ROCC. The idea here is to provide data on 
exercise participants, not capable of being tracked by ACMI, to the 
ACMI via the LRR-ROCC/AICU-ACMI interface (see attach. 1). 
The results: a more complete, accurate air picture, providing an 
added measure of flight safety both to the military and civil sectors. 


The statements that there exists an inability to automatically insert flight 
plan data as well as track data on aircraft not equipped with ACMI pods into the 
3DS is an attempt by the 11 ACW to provide evidence to support the RDMS as a 
solution. A requirement to automatically fuse data derived from outside C3 
systems may exist, but the experiment was not designed to support this question. 
Furthermore, no evidence exits in the Report to support this requirement. 

The integration issues also address the important safety role the radars 
play in keeping track of aircraft not equipped with ACMI pods. Aircraft not 
equipped with ACMI pods are not tracked by the ACMI and the radars provide 
the only measure of safety. While this is true, and there is a requirement to 
integrate this data, the report claims an added measure of safety 1t can not support 
with data obtained by the experiment. The experiment was not designed to 
support a flight safety issue. Therefore, conclusions about flight safety can not be 


extrapolated from the experiment that was conducted. 


4] 


The next factor addressed (paragraph e) is the communications interface 


between the ROCC and the radar. 


e. The communication interface between the LRR and the 
ROCC needs to be made more efficient. At present massive 
amounts of radar data, most of it useless clutter is being bulk 
encrypted and sent over the communications circuits to the ROCC 
(see attach. 2). Sending only true target information, filtered from 
the clutter, will reduce the burden on the interface, thereby making 
this interface more efficient. 


Again, this a valid requirement, but one that is not addressed by the 
current Plan. In order to make any claims about how much burden would be 
reduced by an RDMS, experiments have to be designed with this question in 
mind. 

The report states (paragraph f) that one of the factors is: "Too much 
useless data is being sent to the ROCC computer for processing. Sending only 
data which contains true target information will inherently make the ROCC 
computer more efficient.” How can this statement be verified from the design of 
the experiment put forth in the Plan? Again, a separate experiment must be 
conducted if the 11 ACW wants to verify these claims. 

Human variability is addressed in paragraph (g). The Report says: "A 
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human variability in the target detection and tracking process exits. " There is no 
denying that variability exists between individuals in the tracking process. 
Furthermore, it would seem likely that if the RDMS lived up to its capability the 
variability in the tracking process could be reduced. However, the Report sites 
one of its attachments (see attachment 3 in Appendix D) as supporting evidence 


to this claim. This attachment is a view graph of a radar scope demonstrating the 


difference in the amount of data between an ATS equipped radar and a non-ATS 
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equipped radar as seen from an operator perspective. This view graph does not, 
however, support the claim that variability in the tracking process can be 
reduced. Visually looking at the differences does not tell the whole story. The 
view graph does not provide insight as to how the data is being processed by the 
ATS or whether the ATS is eliminating targets of interest (the primary objective of 
the Plan). A statistical analysis of the raw data needs to be accomplished 
according to a design like that laid out in the new Plan. 

The final factor the Report mentions is the need to integrate all of the 
stovepipe C2 systems in the theater (paragraph h). The reports says: "Systems, 
1.e., AICU, ACMI, CTIS [Commander's Theater Information System]/3DS, need to 
be integrated more efficiently. Information available to one system can be made 
available to another system. For example: track data the AICU receives from an 
LRR can be made available to both 3DS and ACMI." Again, this is clear 
evidence of the shift in emphasis between the Plan and the Report. The emphasis 
has shifted from validating the ATS's capabilities to using the Range 
Improvements Project requirements as a means of justifying the procurement of 
the RDMS. 

The fifth and final section of the Report is the Discussion Section. The 
material mentioned in this section could either be totally eliminated from the 
report or included as part of the Introduction Section, providing background. 

The general impression of the Discussion Section is that the RDMS is 
the "catch all" single solution to all of the Range Improvements Project's 
requirements. The introduction paragraph states: "A solution will be offered, 
tested against criteria, and shown how this solution will solve the problem." The 


report argues that if the RDMS can hold up to the scrutiny of criteria that are 
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established for the first time in the Report, then the RDMS must be the a valid 
solution. 

The entire discussion appears to be a sales pitch to the decision makers 
on the importance of the RDMS to the Range Improvements Project. The Report 
essentially ignores the original purpose of the demonstration, as stated in the Plan 
or otherwise, and argues that the RDMS is the only solution because it meets all 
of the criteria the Report established. The Discussion Section makes no attempt 
to address the issue of whether or not the claims made by the vendors about their 
equipment are valid. The only things offered are excerpts from the contractors 
final reports on how great their equipment is and all the reasons the 11th AF 
should buy their equipment. There is no independent, unbiased, scientific 
analysis of the data. The only analysis is from the vendors, using their equipment 
and their analysis tools. 

The 11th ACW is asked to have faith that the analysis conducted by the 
vendors, using their own equipment, is reliable. The demonstration started out 
with the stated purpose of determining the validity of the ATS's capabilities and 
ended up using the ATS as a solution to an integration problem. This dnft can be 
traced back to the lack of focus of the Plan, which did not provide a "road map" 
that the 11t1 ACW staff could follow. This lack of focus lead to a Plan and 
Report that lacked credibility and do not provide concrete answers for decision 
makers. This is probably why the PACAF has yet to acquire a RDMS for the 
Alaskan radars. 

2. Solution (Introduction) 

The Introduction section in the final report provides information to the 

reader of the report who has not read or had access to the Plan. Therefore, 


everything mentioned in the Introduction Section of the revised Plan should be 
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included in the final report, plus any additional information revealed during the 
experiment. 

During the time of the demonstrations, the Alaska 2000+ Range 
Improvements Project became a prime interest of the 118 AF commander. The 
11th ACW saw the Range Improvements Project as a funding vehicle for their 
RDMS project. The background politics explain the discontinuity between the 
Report and the Plan; however, this is not an excuse for allowing a shift in the 
problem emphasis. The RDMS project can only suffer. When the focus shifts 
back to the RDMS, questions will be asked about the validity of its claimed 
capabilities. If both the Plan and Report focused on the questions of validating 
RDMS capabilities, then the 11th ACW would have strong supporting evidence 
for piggybacking on the Range Improvements Project. 

3. Mistakes (Experimental Design) 

Like the Introduction Section, the Experimental Design Section shares 
similarities between the experimental plan and report; the difference is that greater 
detail must be included in the report. It should explain the experimental setup 
including the physical setup, who the test subjects or experimental units are, what 
special equipment is required, and give the trial schedule. The hypotheses that 
are to be examined should be stated specifically, as should any assumptions that 
will be made. The statistical design of the experiment should be explained, along 
with definitions of measures that will be employed and instrumentation that will 
be used to collect the data. Provisions should be made for testing the setup and 
design, and for the trials. 

The Report fails to adequately address the experimental design. The 
report contains Assumptions and Criteria Sections that could be considered 


elements of the experimental design section. The assumptions deal only with 
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integration. The assumptions are: “Industry has the capability to solve the 
integration requirements we have established during this study. The AICU will 
be able to receive track messages from the RDMS and route these messages back 
and forth between the 3DS and ACMI." However, assumptions used to analyze 
the data collected from the experiment are not mentioned. 

The Criteria Section deals only with requirements that have to be met in 
order to provide an adequate solution to the interface problems. Each of the five 
criteria are biased towards the RDMS as a solution. The report lists the following 


as criteria: 


a. The solution must be able to reduce the burden on the LRR-ROCC 
interface. 


b. The data that's being output is in a format that can be used by the 
ROCC computer as well as the AICU. 


c. The solution must be able to integrate and interface the AN/TPS-63 
radar data in the correct format into the ROCC where it will be routed via the 
AICU to the 3DS and ACMI. 


d. The system must have an open architecture design, being able to be 
incorporated into future systems (ROMS/RAMMS). 


e. Versatile user friendly workstations, providing self diagnostics and real 
time radar status monitoring, as well as GCI control. 


Clearly each of the criteria takes advantage of the proposed benefits 
touted by the vendors of the RDMS equipment. Once again, the political 
pressures are at work: If the RDMS could meet all of the criteria, how could the 
decision makers ignore the RDMS and not allow funding to a system that directly 
impacts the Range Improvements Project? 

4. Solutions (Experimental Design) 

The report should address the experimental design in same manner as 

the Plan. The experimental design section tells the reader how the experiment 


was carried out. No details should be left out. The report must provide details 
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conceming the setup of the experiment, hypotheses must be clearly stated (these 
are the specific research questions), assumptions made must be listed, the 
Statistical design of the experiment must be presented, the issue of what kind of 
data was collected, and how it 1s to be collected, must not be overlooked, and 
testing and pilot trials have to be mentioned. 

5. Mistakes (Data Becenintion) 

This section of the plan should contain an example of the raw data, (the 
entire set of collected data should appear in an appendix) problems experienced 
with collecting the data should be addressed, and the data should be presented in 
an understandable and easy to read format (i.e., tables or spreadsheets). Any 
methods used to reduce the data, or other manipulations, should be explained. 

The Report does not even mention the data. There is little doubt as to 
why the incorrect data was collected because neither the Plan nor the Report was 
concerned about data collection. 

6. Solution (Data Description) 

All of the details described in the revised Plan need to be included in the 
final report. As a matter of fact, the report should contain even more detail than 
the plan. The report should contain an example of the raw data, and problems the 
data should be addressed. All of the data collected should be presented in an 
appendix or in an archive. 

Readers may want to replicate the experiment to verify the results 
claimed in the report. The data coding scheme (an example appears in the revised 
Plan) must be included. This will allow the reader to decipher the way the 
experimenter coded the data, and it will aid in understanding the analysis. If 
possible, and for the sake of clarity, the data should be presented in a tabular 


form. If the data is manipulated or reduced in any way, this has to be addressed. 
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The bottom line: The data description must be detailed and accurate enough so 
that the experiment can be replicated and the results verified. 
7. Mistakes (Analysis) 

The Analysis section 1s probably the most important section of the 
report. This section should contain a summary of the analysis plan that was 
presented in the experimental plan, a detailed methodology of how the analysis 
was conducted, the results of the analysis, and any additional assumptions that 
were required. 

The Report lacks any significant analysis section. No statistical analysis 
of the data is accomplished. The view graph attachments at the end of the report 
(see Appendix D) are the only things presented as the analysis. Basically, the 11 
ACW presented the findings of the vendors. No attempt was made, nor was any 
data provided, to verify the claims made by the vendors. The problems with the 
analysis stem all the way back to the lack of focus and clearly defined objectives 
in the Plan and the resulting inability to maintain focus from the inception of the 
Plan through its execution. 

8. Solution (Analysis) 

Data that can provide the answers to specific research questions, 
generated from well defined objectives, is a key piece of the solution to the 
analysis failure. There are several elements that are essential to the analysis 
section. The analysis should contain a summary of the analysis plan presented in 
the initial experiment plan. The methodology used to collect the data and how 
that data was analyzed must be detailed, and any additional assumptions that the 
experimenter made have to be addressed. The results of the analysis are the heart 


of the entire experimental process. They results allow conclusions to be drawn 
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and recommendations to be made that are objectively based on actual 
experimental outcomes. 
9. Mistakes (Conclusions) 

The Conclusion Section is unique in that it only appears in the 
experimental report. This section presents the results of the hypotheses that were 
listed in the plan, as well as any other interpretations drawn from the analysis of 
the data. These results are then related to the real world problem the experiment 
set out to answer. An overall summary of the experiment should also be 
presented in this section. 

The Report states: "... demonstrations conducted by Litton Data 
Systems, Sensis Corp., and GE Syracuse, have clearly shown that industry possess 
the capability to meet our established criteria. A RDMS at the LRR would 
integrate radar data from both the AN/FPS-117 and AN/TPS-63 [the proposed 
Westinghouse radar that would be acquired to enhance radar coverage in the 
expanded training areas] radars into the ROCC, 3DS, and ACMI while at the 
same time increasing the efficiency of the entire Alaska 2000+ Range network." 
First, the 11 ACW had little or no involvement with the GE demonstration, so it 
will not be addressed in this analysis. Second, the motivation behind the report 
was to show how the RDMS equipment could be used to integrate the various 
in-theater C2 systems. Clearly a different objective than what was initially 
proposed: To verify the claims made by the vendors about their equipment 
capabilities. Since the report argues for the integration benefits of the RDMS 
equipment, the analysis presented in the attachments is arguing in favor of the 
capabilities. In short, the conclusions are not valid. There is no evidence to 


support the conclusions being drawn. 
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10. Solution (Conclusions) 

The main ingredient to solving the problem with the conclusions is 
maintaining a constant focus on the objectives of the experiment throughout the 
Plan and Report. One approach to achieving this focus is the use of the model 
and paradigm. The model contains four elements in the Conclusion Section. The 
first element is the hypotheses results, along with the interpretations of these 
results. The second element is any other interpretations. Third, and probably the 
most important, the real world meaning of the results. Finally, an experimental 
Summary should be drawn. According to the paradigm, the conclusions are 
",..supportable if they are valid results of an analysis that follows logically from 
the study.” 

11. Mistakes (Recommendations) 

The report should finish with a Recommendations Section. This section 
should contain a discussion of potential changes to the experiment along with 
recommendations for further work in this area. 

The Report does not contain the elements described above, instead the 
Report makes a recommendation in favor of the RDMS. Based upon all of the 
analysis of this Report, there is no evidence to support this recommendation. 

12. Solutions (Recommendations) 

The Recommendations Section of the report, in addition to offering 
recommendations, should contain sections dealing with changes to be made to 
the experiment. Additionally, the recommendations section should address any 
requirement for further work in this area. 

13. General Issues 
Finally, there are two general issues concerning the overall quality of the 


experiment. First, is the issue of simplicity. According to the paradigm, the report 
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is "...simple if it is short and illustrative, has a minimum number of assumptions, 
alternatives, hypotheses, and measures, plus easily understood analysis and 
conclusions.” The Report fails to meet the definition of simplicity. While it is 
short, the Report fails to illustrate how the experiment was carried out. The 
assumptions are not representative of an experiment initially designed to verify 
the capabilities of the ATS. Instead of listing specific hypotheses, integration 
criteria are listed. Neither measures or an analysis is addressed by the plan. A 
marginal attempt at an analysis is made using view graphs. Finally, the 
conclusions stated apply to benefits of integration and not about RDMS 
capabilities. 

The second issue addresses the consistency of the report. “The study is 
consistent if there is compatibility in scope and detail at each step. In addition, 
direct linkage between the steps must maintain continuity with the objective.” 
The shift in the objectives between the Plan and Report is evidence the study is 


not consistent. There is no continuity between the Plan and the Report. 


C. CONCLUSIONS 

The ATS Demonstration Plan was analyzed using the Experiment Plan and 
Report Formats as a model of how a plan is organized. The Evaluation of Studies 
paradigm provided definition to the organizational model. Highlighting the 
mistakes of the Plan was only part of the analysis. The next step was to take the 
lessons learned from the Plan and apply them to a new hypothetical plan. 

The analysis of the Report (known as the LRR STAFF STUDY) focused on 
the analysis and recommendations. Like the analysis of the Plan, the Report 
analysis utilized the model and paradigm as evaluation tools. The Plan and Report 
are similar because most of the material covered in the Plan is summarized in the 


Report. Unlike the analysis of the Plan, the analysis of the Report does not 
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provide a new Report because data is missing from both Litton and Sensis. The 
missing data can not simply be made up in order to complete an analysis 
consistent with the new Plan. 

The analysis reveals many mistakes in both the Plan and Report. However, 
many lessons can be leamed from this experience. These lessons have been 
applied in the creation of a new Plan. Furthermore, generic lessons, applicable to 


experiments and experimenters, are found in Chapter IV. 
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IV. LESSSONS LEARNED 


Specific lessons can be learned by studying the Plan and Report described in 
this thesis. The analysis chapter illustrated many mistakes that can be made 
during an experiment. This chapter takes the specific mistakes made during the 
RDMS experiments and creates a list of lessons learned. Additionally, the lessons 
the author (who was a major player in the original experiment) learned are listed 
and are applicable to experiments in general Hopefully, these lessons learned may 
serve as a guide to staff officers that have little experience in conducting 
experiments but are tasked to conduct an experiment in an operational 


environment. 


A. SPECIFIC LESSONS LEARNED 

Experimental plans and reports have specific roles. The plan describes in 
detail how the experiment will be executed. The report describes the execution 
of the experiment, providing an analysis, drawing conclusions and making 
recommendations. The lessons described in this section all relate to specific 
mistakes identified in the Plan and the Report. 

1. Clearly Defined Objectives 

Clearly defined objectives are a must. They provide the foundation 

upon which the entire experiment is based. For example, the objectives defined in 
the experimental plan provide the focus for the entire experiment. Without this 
focus the researchers may find it very difficult, if not impossible, to carry out a 


successful experiment. 
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2. State the Purpose 
State the purpose of the experiment, preferably in an Introduction 
Section, both in the plan and in the final report. The purpose should describe the 
real world problem the experiment is trying to solve along with specific questions 
the experiment seeks to answer. This is important because it helps to maintain the 
focus for the experiment. After this groundwork is laid, a description of the 
approach the researchers will use to try and solve the problem must be addressed. 
Finally, to help build a common mental picture of what the experiment hopes to 
achieve, there should be a statement about the anticipated results of the 
experiment. a 
3. Define the Scope of the Experiment 
The importance of this experiment, why the experiment is being 
conducted, and what impact will it have, constraints placed on the conduct of the 
experiment, and boundaries on the applicability of the results, should be spelled 
out in a section that describes the scope of the experiment. 
4. Provide a Detailed Experimental Design 
The experimental design is the heart of the plan. This section should 
provide the details of how the experiment is to be executed. It should be detailed 
enough so that when the experiment is verified it can be replicated. 
5. Provide a Data Description 
What kind of data is going to be collected and used for analysis? Will 
an analysis of the data provide answers to the questions being asked? These 
questions must be answered prior to the start of the actual experimental trials. 
The researcher must know what kind of data is being collected and whether or 


not it will be useful. 
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6. Prepare an Analysis Plan 
After the data is collected, it must be analyzed. To prevent bias in the 
analysis, and to ensure that the nght data is collected, the researcher must have a 
plan of how the data will be analyzed. While the analysis plan is stated in the 
experimental plan, the actual analysis and results are the crux of the final report. 
7. Conduct Testing and Pilot Trials 
A preferred method of determining whether or not the correct data will 
be collected during the experiment is by conducting testing and pilot trials prior 
to the actual execution of the experiment. Testing and pilot trials prior to the start 
of the 11th ACW experiments would have determined that the data being 
collected was incomplete and would not adequately provide for an analysis that 
would yield insights as to the capabilities of the RDMSs. 
8. State Supportable Conclusions 
The results of the analysis should answer the specific questions being 
asked. The conclusions must be supported by the results of the analysis. 
9. Make Sound Recommendations Based Upon Analysis of the Data 
The recommendations that are made should be supported by data 
collected through the execution of the experiment and not by circumstantial 


evidence. 


B. GENERALIZED LESSONS LEARNED 

The generalized lessons learned are subjective lessons the author developed 
during the course of this thesis. These lessons, hopefully, will serve as a guide to 
those staff officers with little background in experimentation, but are nonetheless 


called upon to conduct experiments in an operational environment. 
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1. Ask Questions 
Can experiments be conducted in an operational environment? 
What will the operational impact be if the experiment is conducted? 
Will the operational mission adversely affect the experiment? 
Will experiments conducted in the operational arena yield useful results? 
Should commander involve the staff to conduct these experiments? 
Is the staff knowledgeable enough to undertake this task? Or will outside 
assistance be required? 
2. Gather Knowledge Base 
Ensure knowledge exits within the staff to conduct such an experiment. 
If necessary seek help from other sources. Maybe even have some other unit or 
agency come in and conduct the experiment. This will alleviate any bias the 
home staff may have towards the experiment. Two views existed: one favoring 
the ATS equipment, and one wanting no part of the equipment. Interestingly, the 
lines of opposing opinions were drawn between operations and maintenance 
staffs. The operations staff wanted to see the ATS equipment incorporated while 
the maintenance staff wanted to continue on with the current ECP fixes. 
3. Be Open Minded, Yet Remain Cynical 
Don't get locked into one solution and fail to seek other viable options. 
Be cynical, ask hard questions. Don't be impressed with "bells and whistles.” 
Seek proof of claims from independent sources. 
4. A Good Plan is a Must 
A good plan must have clearly defined objectives. These objectives are 


then broken down into questions the plan will be designed to answer. 
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5. Keep Research Questions Simple 
Keep research questions simple and in line with operational constraints. 
Trying to solve complicated research questions in a constrained operational 
environment may prove to be too difficult and best left to the laboratory. Break 
the complicated research question down into smaller pieces. 
6. Pilot Trials are Essential 
Pilot trials ensure the data collected will provide answers to the 
questions being asked. Ensure the data is replaceable. 
7. Keep Detailed Notes 
Trying to remember of the details of an experiment is an impossible task 
unless detailed notes are kept. 
8. Do Not be Afraid to Modify the Experiment 
If the pilot trials indicate flaws with the experiment, change whatever 
you need to change. Objectives may have to be modified. Whatever the case, do 
not be afraid to admit mistakes. The important thing is to correct the mistakes 
before the actual trials begin. Time and money are precious commodities. If 
mistakes remain undiscovered or overlooked, rerunning the entire experiment 
may not be feasible. 
9. Use the Right Statistical Tool for the Right Job 
Using certain statistical tools requires different assumptions to be made. 
Using the wrong tool seriously undermines the conclusions drawn from the 
analysis. 
10. Report Results in a Clear, Concise, and Simple Format 
Do not attempt to give a technical brief to a commander and his staff. 


You could quickly find yourself out in the hall. If, by chance you are allowed to 
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proceed, you run the risk of creating a great deal of confusion and seriously 
jeopardizing an credibility of your office. 
11. Present Only the Facts, Don't Speculate 
When asked to report the results of your analysis report only the results. 
If you do not have the answer say so and do not interject personal opinions. 
Doing so only clouds the results of the experiment. 
12. Be Careful About Extrapolating Results to Other Situations 
This lesson is a corollary of the above lesson. There is an overwhelming 
urge to extrapolate the results of experiments to other areas of interest. Care must 
be taken. The results of the experiment are for a certain set of conditions. 
Extrapolating results of an experiment to a different set of conditions without 
further research is not wise. In other words, stay within the scope of the 
experiment. 
13. Ensure Consistency Between the Plan and the Report 
The conclusions presented in the final report should be consistent with 
the primary objective stated originally in the plan. If, for some reason the 
objectives of the experiment change, then the plan should reflect the changes 


prior to conducting the experiment. 
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V. CONCLUSIONS 


The data collected could not provide the answers to the primary questions, 
and that analysis of the Plan and Report revealed several weaknesses which no 
doubt contributed to this failure. A Summary of the deficiencies is given in the 
following paragraphs. 

The overarching problem with the Plan was that it failed to clearly define the 
objectives of the experiment. This resulted in a lack of a clearly defined purpose. 
As a result, the Plan constantly strayed from the primary objective: venfying the 
ATS's capabilities. 

The Plan also failed to provide a detailed description of the expenmental 
design. As it is presented, the Plan does not describe how the experiment will be 
conducted to verify the ATS's capabilities. This resulted in procedures calling for 
experiments that did not relate to the primary objective. 

The lack of a clearly defined data description resulted in an inability to 
conduct a thorough analysis. The Plan states that data will be collected and then 
analyzed. However, there is no description of the data, how it will be collected, or 
any testing done to determine whether or not the data is the correct data needed 
for the analysis. The analysis could not be conducted because insufficient data 
was collected. 

The Plan also lacks an analysis section. There is no mention as to how the 
data will be analyzed. The result is seen in the Report where the analysis uses 
subjective arguments to support certain criteria and view graphs showing the 


benefits in reducing the amount of data being sent by the radars. 
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The lack of clear objectives also plagued the Report. As a result, the entire 
emphasis of the Report shifted from verifying RDMS equipment capabilities to 
integrating other in-theater C3 systems. 

The Report lacked any significant analysis. Instead of analyzing collected 
data to test the validity of the ATS, a set of criteria, not previously discussed, was 
presented. The analysis consisted of examining whether or not the RDMS 
equipment would meet the criteria. As a result, little if any, information could be 
gleaned from the analysis. 

The conclusions presented in the Report were not supported by a Statistical 
analysis from data collected during the experiment. Instead, the Report presents 
view graphs providing a visual representation of the RDMS equipment 
capabilities. The result is that there are still questions about the validity of the 
RDMS and its capabilities. The conclusions seemed to be based upon 
circumstantial evidence not on evidence provided by the analysis data collected 
during the experiment. 

The final result of the Plan and Report is a positive endorsement of the 
RDMS as a solution to the radar data management problems found in Alaska. 
However, based upon the analysis, the recommendations made in the Report are 
of doubtful validity. 

The model used in this thesis is an accepted evaluation tool. There is no 
requirement that all experimental plans and reports follow this model; but, it is the 
opinion of this author that the Plan and Report failed partly as a result of not 


following a model such as the one taught at the Naval Postgraduate School. 


60 


VI. SUMMARY AND RECOMMENDATIONS 


The purpose of this thesis was to analyze an actual experimental plan and 
report and provide lessons learned. The entire experimental process, starting with 
the plan development and proceeding through the publication of the final report, 
lasted approximately two years. The motivation behind this thesis was to 
illustrate the difficulties involved with experimentation. Compounding these 
difficulties is the fact that this author, who was in charge of the experiment, 
lacked the proper training needed to carry out an experiment successfully. 
However, after attending the Naval Postgraduate School (NPS) and studying the 
correct way to conduct an experiment, this author decided to use the lessons 
learned at NPS to analyze what was done two years before. 

Chapter II provided background stating the motivating factors behind the 
experiment. Chapter III is the heart of the thesis. This chapter analyzed the ATS 
Demonstration Plan and the LRR Enhancements Staff Study. Mistakes were 
pointed out and solutions were offered based upon the Experiment Plan and 
Report Format as well as the Evaluation of Studies paradigm presented in the 
Joint Command and Control and Communications curriculum offered at the Naval 
Postgraduate School. Chapter IV provided the lessons learned from the 
experiment. The purpose of this chapter was to provide a guide to those staff 
officers who have little experience in experimental design and analysis but are 
nonetheless tasked to conduct an experiment in an operational environment. 
Hopefully, the lessons learned may serve as a guide so that these staff officers can 
provide reliable recommendations to decision makers. 

Further work needs to be done to determine the validity of the RDMS. 


Experiments need to be conducted that determine whether or not the RDMS 
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device 1s eliminating data on targets vital to air defense and air combat training. 
After this 1s determined, questions of integrating this system into current C3 


architectures can be addressed. 
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APPENDIX A 


GLOSSARY 
ACMI 

ACMI stands for Aircraft Combat Maneuvering and Instrumentation. The 
ACMI was initially designed as a training device for fighter pilots. The ACMI 
provides a three dimensional view of aircraft in relation to its environment 
providing lessons learned for mistakes made in flight. 
AICU 

AICU stands for Advanced Interface Control Unit. This system is designed as 
a translator and router of data. The sia of the AICU is to serve as a main 
point where data from different sensors can be interchanged. Data from different 
sensors utilizing one type of protocol can be exchanged with data of other 
sensors utilizing a different protocol. For example, data from a Radar 
Management device located at a radar can be sent to the AICU translated into an 
appropriate protocol so the data can be displayed on the 3DS as well as on the 
radar consoles of the ROCC. 
CTIS/3DS 

CTIS/3DS stands for Commander's Theater Information System/Digital Data 
Display System. CTIS is a network of computers designed to collect and fuse 
data. The fused data is then displayed using a large screen display. The computer 
hardware and software required to project the fused data is called the Digital Data 
Display Device (3DS). 
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GCI 

Ground Control Intercept. Air weapons control officers station in a ground 
based radar unit with the task of controlling aircraft. 
LRR 

Long range radars. In Alaska the LRR is the AN/FPS-117 Minimally 
Attended Radar. 
Plot 

Data necessary to describe a point in space. A two dimensional view (X and 
Y axes) 
Target 

Consists of more than one plot. Gives the third dimension. Associated data: 
Heading, Speed, Altitude. A direction can be determined from a series of plots 
called a target. 
Track 

A track is a target with alphanumeric data (symbology) attached. The 
symbology can either be manually or computer derived. Tracks are established 
by a radar operator who determines whether or not a target represents a valid 
aircraft. The operator has a direct involvement in the decision making process. 
ROMS/RAMMS 

Remote operational maintenance system designed to provide long distance 


status monitoring of the radar. 


65 


APPENDIX B 


ADVANCED TRACKING SYSTEM DEMONSTRATION PLAN 
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FROM: 11 TCW/DO 
6900 9th ST STE 301 
ELMENDOREF AFB AK 99506-2270 


SUBJECT: Advanced Tracking System Demonstration Plan 
TO: SEE DISTRIBUTION 


|. Forwarded herewith is the Advanced Tracking System Demonstration Plan, | 
Oct 91. This demonstration is being conducted at the direction of 11 AF/CC. 


2. This plan outlines those actions necessary to demonstrate the capabilities and 
utility of the Advanced Tracking System in the Alaskan Theater. 


3. This plan is designed to demonstrate the operational capabilities of the 
Advanced Tracking System as part of the current Alaskan Surveillance and 
Command and Control System. Of particular interest will be the Automatic 
Tracking System's ability to reduce demands on the ROCC's central computer ( 
processing time and memory space), restore the FPS-117, Long Range Radar, 
performance to its original operational specifications for target detection, and to 
reduce human variability in the detection and tracking process. The 
demonstration will take place from 7 Oct 91 to 18 Oct 91. The equipment will be 
located in the ROCC in order to take advantage of the existing communications 
circuits feeding radar data from the Long Range Radars (LRRs) to the ROCC 
computer. Locating the ATS in the ROCC will allow access to any radar without 
the logistics of transporting people and equipment to remote sites and make the 
equipment available for maximum visibility. 


GEORGE A. PAHLS, Colonel, USAF 1 Atch 
Deputy Commander for Operations Advanced Tracking System 
Demonstration Plan 
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PLAN SUMMARY 


I. PURPOSE: To outline responsibilities and procedures for the Advanced 
Tracking System Demonstration of 7 Oct-18 Oct 91. 


2. CONDITIONS FOR IMPLEMENTATION: This project plan will be the basis 
of the Advanced Tracking System Demonstration. It will be implemented at the 
direction of the 11 TCW/DO. — 


3. OPERATIONS TO BE CONDUCTED: 


a. Forces Assigned. All tasked units will respond when this plan is executed 
unless specifically excused by the implementing authority. 


b. Supporting Plans. Not required. 


4. OPERATIONS SECURITY (OPSEC): In the execution of this plan, OPSEC 
must be a matter of continuing concern in order to minimize disclosure of 
sensitive information. Since operations may reveal limitations of the Automatic 
Tracking System, the AN/FPS-117 Long Range Radar, and the ROCC AN/FYQ- 
93 computer, activity must be conducted so as to preclude inadvertent disclosure 
of sensitive and/or classified information. 


5. COMMAND RELATIONSHIPS: Normai. 
6. LOGISTICAL APPRAISAL: All logistical matters envisioned are feasible. 
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iPUIRIPOSE 


|. The stated purpose of the demonstration is determine whether or not the ATS 
can be used in the Alaskan Theater to improve radar performance and make 
recommendations concerning the desirability of incorporation the ATS in the 
Alaskan Surveillance and Command and Control System. The primary objective 
is to collect sufficient data to determine the ATS’ capability for substantially 
improving the probability of detecting targets (at lower altitudes and clutter 
environment). The ATS, by sending only true target data to the ROCC computer, 
could reduce the amount of data the computer has to process. A significant 
reduction of data will reduce the demands on computer processing time and 
memory/storage space; both of which are nearly saturated. 
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PROJECT WANAGEMENT 


1. In order to manage the demonstration, an Advanced Tracking System Project 
Management Team was established. The demonstration team leader is 11 
TCW/DOXxX; other team members are listed in paragraph 3. 


2. The charter of the project management team is to: set the demonstration 
objectives; develop evaluation criteria; conduct and observe demonstrations, 
record data on the results of the demonstration; evaluate the potential 
application of the ATS; and publish a project report. 


3. The following agencies are participating or providing support for the ATS 
demonstration: 


a. 11 TCW/DOXXQ -- Capt Pierce 

b. 11 TCW/DOP ~— -- SSgt Myher 
c. 11 TCW/LGOR_- -- MSegt DeLuca 
d. 11 TCW/LGKC_ -- MSet Shuler 
e. 11 TCW/LGKM -- TSgt Cushman 
f. 744 ADS/DOO ~ -- Lt McNeil 

. 21 St TFW/DOW -- Capt Hill 


GQ 
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SYSTEM DESCRIPTION 


]. The AN/GYQ - 51, Advanced Tracking System, manufactured by Litton Data 
Systems is a State of the art tracking system, providing sensor interoperability 
between centralized, decentralized, and autonomous air surveillance systems. The 
ATS is a front end processor located at the radar, intelligently filtering true targets 
from clutter and sending only actual track data to the command center for 
display. This capability allows larger volumes of actual track data to be 
transmitted over communications channels without resorting to desensitizing or 
blanking of the radar. 


2. The ATS functions by detecting, establishing, and displaying tracks without 
operator intervention. The ATS accomplishes these functions through a four step 
process: 


considers all radar returns via scan to scan processing; 
. identifies true targets; 

establishes tracks using true targets; and 

. maintains tracking continuity using predictive logic. 
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DEMONSTRATION OBJECTIVES 


Purpose: 


There are four objectives of the demonstration. The overall purpose of each is to 
show how the Alaskan Theater's surveillance and command and control 
environment can be enhanced by the Advanced Tracking System. 


Objectives: 


1. Reduce demands on the FYQ-93 ROCC central computer; processing time and 
memory space. At present, central computer processing time and memory storage 
Space is nearly saturated; while memory storage in the processor controller is 
overloaded to the point it "dumps" data. 


2. Radar Enhancement: Regain loss of probability of detection, sacrificed at the 
expense of clutter reduction solutions to the radar. With the ATS’s ability to 
extract true targets from clutter, the radar beam could be lowered thus increasing 
the probability of detection of low level targets. 


3. Increase probability of detection by reducing human variability in the 
detection/tracking process. The ATS serving as a front end processor at the radar 
will automatically detect, establish, and pass target data to the Region Operations 
Control Center (ROCC). The presentation of only true targets in the ROCC will 
eliminate operator decisions concerning the validity of targets. This will increase 
the probability of establishing air tracks. Also because only true targets are 
presented, maintaining track continuity will improve for both computer tracking 
algorithms and when operators override computer tracking. 


4. Role of the ATS in Alaska: By analyzing the data collected during the 
demonstration, we will draw inferences concerning the utility of the ATS in 
Alaskan Theater. Areas of interest: advantages of incorporating the ATS at each 
AN/FPS-117 in Alaska from a sensor interoperability standpoint, added measure 
of safety in large scale exercises. 
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Procedures 


1. The following procedures serve as a guide to ensure successful completion of 
the four objectives for the demonstration. 


2. Three radar sites (Cape Romanzof, Kotzebue, and Tatalina) will be used to 
collect data to satisfy objectives one and two (reduce demands on the FYQ-93 
ROCC central computer, and radar enhancement). These sites were chosen 
because they possess the latest clutter reducing software (i.e. ECP 175 and the 
Multiscan Detection Processing (MDP)). At different times each day, data will be 
collected from each site for a period of 105 minutes. Each fifteen minutes, within 
the data collection window, the radar will be placed into a different configuration 
(e.g. ECP 175 enabled and MDP disabled). Data will be collected via data 
reduction, later collated on the Site Data Sheets (figure 1) and analyzed. 


3. Data will be collected using low level flights Gif available) in the ACMI range. 
This data will be used to assess the ATS's ability to enhance the probability of 
detection (Pd) of the 

FPS -117 radar. 


4. A tracking test will be used to meet the third demonstration objective (increase 
probability of detection by reducing human variability in the detection/tracking 
process). A operator will be selected at random and given an area of 
responsibility (AOR) to initiate and maintain track continuity on all tracks in the 
AOR. The operator will operate with and without the benefits of the ATS and all 
track histories will be recorded via data reduction. A comparison will be made of 
the data collected from the data reduction products to determine whether or not 
an operator will realize an increased probability of detection by reducing human 
variability in the detection/tracking process. Detailed instructions will be 
provided to each surveillance operator by the project coordinator--thus insuring a 
degree of procedural standardization. 


5. In order to meet objective four, the potential role of the ATS in Alaska, 
inferences will have to be drawn based upon the data collected in objectives one 
through three. 
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THE ATS DENMIONSTRATION LOG 


The purpose of the ATS Demonstration Log is to ensure accurate and timely 
recording of data during the demonstration. 


The ATS demonstration log shall contain the following information: 


a. Site 
b. Start time 
c. End time 
d. Radar configuration: 
1. ECP 175 (Anomalous Propagation) activated (beam raised or 

lowered). 

2. MDP "ON" or "OFF". 

3. Data count without the ATS (search only). 
. Data count without the ATS (SIF only). 
Total data count without the ATS (Search and SIF). 
. Data count with the ATS (search only). 
. Data count with the ATS (SIF only). 
. Total data count with the ATS (Search and SIF). 


. Number of tracks without the ATS. 
10. Number of tracks with the ATS. 
e. ACMI data for comparison. 


f. Record of ROCC track histories for comparison, data reduction. 
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Data Products 


Data products required to ensure accurate analysis. 

1. ATS demonstration log. This log will be maintained at the Air Surveillance 
Technician position. 

2. Data reduction, hard copy print out of flights within the ACMI. Flights will 
be recorded on tapes located at the ACMI. Dunng playback, at the ACMI, hard 
copies of flight data will be made. 


3. Data reduction from the FYQ-93 (ROCC) computer will be accomplished by 
11 TCW/DOP covering the times of the demonstration. 
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ATS Demonstration 


system Trouble Log 


The System Trouble Log documents any difficulties experienced during the 
demonstration. 


Date: 
Operator 


Synopsis of Event: 
(if needed, attach additional sheets) 


Corrective Action: 
(If needed, attach additional sheets) 
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ATS Demonstration 


Personnel Familiarization Forms 


The Personnel Familiarization Forms document equipment familiarization received 
by the Litton Data System representatives. 


Litton Data System Representative: 


Military Personnel Trained 


Name Date Time 
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DATA AUTOMATION SUPPORT IREQUEST 


The Data Automation Support Request form enclosed is a form developed and | 


used by the 744 ADS to request data reduction. 
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ITEM 


Advanced 
Tracking 
System 
(ATS) 


Tactical 

Display 

Console 
(TDC) 


Work space 


ROCC computer 
ports 


RESOURCES 


FUNCTION 


Tracker 


Display for ATS 
output 


ROCC 


Data for ATS 
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SOURCE 


Litton Data Systems 


Litton Data Systems 


USAF 


USAF 


ADVANCED TRACKING SYSTEM DEMONSTRATION TIME 
ILIDNIE 


1 OCT 91 


Time: TBD ATS and TDC equipment arrives to the Region Operations Control 
Center (ROCC). 


2-4 OCT 91 
0900- ? Installation and check out of ATS equipment. 
7-9 OCT 91 


Litton Data Systems representatives will conduct demonstrations of the ATS and 
providing familiarization to USAF personnel. 


10 OCT 91 

O800-? From this time forward, until the end of the demonstration, data will be 
collected from various radar sites at random times. This process will be 
accomplished solely by USAF personnel who have received familiarization on 7-9 
Oct 91. 

17 OCT 91 


1700 Demonstration end: Data wiil be collected, compiled, analyzed, with results 
presented in an after action report. 


6-1 


82 


OPERATIONS SECURITY (VU) 


1. (U) General: This section provides guidance for the planning and conduct of 
the ATS demonstration. 


2. (U) The denial of protected information to an opposing force is a command 
responsibility. All commands and individuals associated with any aspect of 
planning, support, or execution of the ATS demonstration plan will safeguard 
operational information in accordance with the guide that follows. 


ESSENTIAL ELEMENTS OF FRIENDLY INFORMATION (U) 


SUBJECT REQUIRING PROTECTION 





System Item 

AN/FYQ - 93 Data Counts 

AN/FPS - 117 Multi scan Detection Processor 
ON/OFF Anomalous Propagation ON/OFF 


Radar Coverage 


AN/GYQ-51 (ATS) Track Detection Data 
NOTE: Taken individually, EEFIls are UNCLASSIFIED. Taken as a whole, they 


are classified SECRET. 
7-1 
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DISTRIBUTION 


11 'h Air Force 


CC 1 
CS 1 
CY. 
DO 5 
XP 1 
LG 1 


ALASKAN NORAD REGION 


CD 1 
11h TACTICAL CONTROL WING 

CC 1 
DO 10 
LG 3 
SC 2 
OTHER 

21 TFW/CC 1 
21 TFW/DO 1 
Litton Data Systems 2 


HARRY J. KIELING, Col, USAF 
Commander, 11!" Tactical Control Wing 


OFFICIAE 


GEORGE A. PAHLS, Colonel, USAF 
Deputy Commander for Operations 
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APPENDIX C 


LRR ENHANCEMENTS STAFF STUDY 
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11% TACTICAL CONTROL WING (PACAF) 
ELMENDORF AFB AK_ 99506-2270 
28 FEBRUARY 1992 





LRR ENHANCEMENTS 
STAFF STUDY 


PACIFIC AIR FORCES 
11TH TACTICAL CONTROL WING 
ELMENDORF AFB ALASKA = 99506-2270 
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FROM: 11 OPG/DOXXQ 
6900 9TH ST STE 301 
ELMENDOREF AFB AK 99506-2270 


SUBJECT: LRR ENHANCEMENTS STAFF STUDY 
TO: 
PROBLEM 


1. The 11 AF needs to acquire a system that will integrate existing systems i.e., 3DS, 
AICU, and ACMI, into a cohesive, efficient system, providing unseen benefits to the 
Alaska 2000+ Range Improvements Project. 


Note: A large proportion of the attachments for this staff study come from the after 
action reports of Litton Data Systems and Sensis Corporation--copied with their 
permission. 


FACTORS BEARING ON THE PROBLEM 
Ze Facts. 


a. Radar data from the AN/TPS - 63 must be integrated into the ROCC. The benefits 
from the additional radar coverage provided by the TPS-63's needs to be disseminated 
and utilized to the fullest extent possible. 


b. Currently, no capability exists to automatically insert flight data on military aircraft 
into 3DS. The NORAD Software Support Facility (NSSF) is working on CSCP # 4569 
which would allow the automatic tell of military aircraft. Hence, creating the pathway 
needed to get data on military aircraft into 3DS. The solution the NSSF is trying to 
achieve is a major Software change: no Solution ts on the horizon. 


c. Track data on aircraft not equipped with ACMI pods, but still players in large force 
exercises (LFEs), are not entered into the ACMI. 


d. Radar data on aircraft not equipped with ACMI pods are detected by LRRs 
Surrounding the range; however, this information is only sent to the ROCC. The idea 
here is to provide data on exercise participants, not capable of being tracked by ACMI, 
to the ACMI via the LRR-ROCC/AICU-ACMI interface (see atch. 1). The results: a 
more complete, accurate air picture, providing an added measure of flight safety both to 
the military and civil sectors. 


e. The communication interface between the LRR and the ROCC needs to be made 


more efficient. At present massive amounts of radar data, most of it useless clutter 1s 
being bulk encrypted and sent over the communications circuits to the ROCC (see atch. 
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2). Sending only true target information, filtered from the clutter, will reduce the burden 
on the interface, thereby making this interface more efficient. 


f. Too much useless data is being sent to the ROCC computer for processing. 
Sending only data which contains true target information will inherently make the 
ROCC computer more efficient. Some of the benefits are: free up available memory, 
decrease the amount of stuff the computer has to process, create greater processor 
flexibility, redistribute the processing workload between PPC-A and PPC-B. 


¢. A human variability in the target detection and tracking process exists. Increased 
reliability of the radar data, see mostly true targets vise mostly false targets, overall air 
surveillance function of the ROCC more reliable and efficient (atch 3). 


h. Systems, 1.e., AICU, ACMI, CTIS/3DS, need to be integrated more efficiently. | 
Information available to one system can be made available to another system. For: 


example: track data the AICU receives from an LRR can be made available to both 3DS 
and ACMI. | 


3. ASSUMPTIONS. 


a. Industry has the capability to solve the integration requirements we have 
established during this study. | 


b. The AICU will be able to receive track messages from the RDMS and route these 
messages back and forth between the 3DS and ACMI. ; 
4. CRITERIA. 

a. The solution must be able to reduce the burden on the LRR-ROCC interface. 


b. The data that's being output is in a format that can be used by the ROCC 
computer as well as the AICU. | 


c. The solution must be able to integrate and interface the AN/TPS-63 radar data in 
the correct format into the ROCC where it will be routed via the AICU to the 3DS and 
ACMI. | 


e. The system must have an open architecture design, being able to be incorporated 
into future systems (i.e., ROMS/RAMMS). | 
| 





f. Versatile user friendly workstations, providing self diagnostics and real time radar 
status monitoring, as well as GCI control. 
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DISCUSSION. 


5. The discussion will take a look at how the Alaska 2000+ Range Improvements 
Project can made more efficient and solve some problems that has been plaguing Alaska 
for years. A solution will be offered, tested against criteria, and shown how this solution 
will solve the problem. 


a. The last several years have seen the proliferation of stand alone systems. <A 
common thread runs through the purpose of acquiring these systems: feed the large 
screen display, essential for making timely and accurate command and control decisions. 
With the shift in emphasis in missions in Alaska, from war time preparedness, needed in 
the cold war, to providing a premier training center for the entire military, these stand 
alone systems must be flexible enough to meet our needs. The purpose of this study was 
to see if there is a way of integrating these stand alone systems into a cohesive network, 
not only for the benefit of the large screen display for the battle commanders, but for all. 


b. The 11 ACW calls the solution a Radar Data Management System (RDMS) (atch 
1). The RDMS is installed at the radar site, and its function is to manage the data coming 
out of the radar sites. A closer examination of the RDMS and how it stacks up to our 
criteria will show the value the RDMS will have in the Alaska 2000+ Range 
Improvements Project. 


c. Reduce the burden on the LRR-ROCC interface is the first criteria the RDMS must 
be measured against. The burden on the LRR-ROCC interface 1s the quantity of data 
having to be transmitted over communications circuits. Most of the data transmitted is 
useless--meaning clutter. The RDMS at the radar will contain software that will 
intelligently discriminate between true targets and false targets or clutter. 
Demonstrations by Litton Data Systems and Sensis Corporation equally illustrated the 
dramatic affect of having a RDMS system at the radar site (See atch 4 & 5). 


d. Sending quality data to the ROCC from the LRR is a great first step. The next 
step is to ensure the data that's being output is in a format that can be used by the 
ROCC computer as well as the AICU. The ROCC computer accepts only plot 
messages;whereas, the AICU accepts only track messages. Again, experience from 
contractor demonstrations has shown there are systems available that output data in 
more than one format. Such is the case with Litton's Advanced Tracking System, and 
Sensis Corporation's Multi-scan Correlator (MSC 2000) (atch 6 & 7). In atch 1, notice 
the RDMS will output plot messages to the Q-93 computer, while at the same time 
outputting track messages to the AICU. 


e. Sending track messages to the AICU allows several things to happen. First, all 
aircraft detected by the radar, sent via track message to the AICU, can then be routed 
directly to 3DS. The interface will solve a long standing problem of how to get aircraft, 
especially military aircraft, automatically into 3DS for display. Second, track messages 
can also be routed by the AICU to the ACMI. The advantages this provides is all 
aircraft on the ACMI ranges not equipped with pods can still be displayed on the 
ACMI. For example, B-1's from a CONUS base fly a strike mission on the range during 
Cope Thunder. In the past, these aircraft would not even be seen on the ACMI; 
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although, they are players. The integration of the RDMS track message to the AICU to 
the ACMI will provide a more accurate and realistic picture. By virtue of being able to 
"see" these types of players, adds an extra measure of safety for both the military and 
civilian aviators. Finally, with the same track messages going to 3DS and the ACMI, an 
automatically updated picture can be seen by viewers of the ACMI and the 3DS. 
Recording and playback capability is the essential point here. Debriefings can occur 
simultaneously at geographically separated units. Litton and Sensis both output radar 
plot and track messages making integration possible today (atch 8). 


f. Integrate the AN/TPS-63 radar into the ROCC is another important criteria. By 
having the TPS-63 radars on the Yukon ranges, will provide low level radar coverage 
never before seen. However, if the data from this radar doesn't go anywhere, the 
benefits will never fully be realized. Putting a RDMS in the TPS-63 will enable radar 
plot messages to be sent to the ROCC computer for display and radar track messages to 
be received by the AICU and routed to the ACMI and 3DS. Increased low level 
coverage, where none previously existed, increased flight safety, increased GCI 
awareness, large screen and ACMI display of low level non podded targets are all 
advantages of having an RDMS at the TPS-63. Both Litton Data Systems and Sensis 
Corporation indicate their systems can be used to integrate the TPS-63 into the ROCC 
(atch 9 & 10). 


g. The RDMS must be a system with an open architecture design, being able to 
incorporate future systems (atch 11). The first "new" system is already becoming a 
reality with the installation this year of the Remote Operator Maintenance System 
(ROMS). 


h. Versatile user friendly workstations are a necessity. The RDMS must have a 
workstation (located at the ROCC) with the versatility of being used to monitor the 
status of the radar on a real time basis. On the other hand, these workstations must have 
the flexibility to be used for either GCI control, or as a recording and playback station. 
The RDMS must have recording and playback features essential to training. Again, both 
Litton and Sensis have demonstrated workstations that could be used (atch 12 &13). 


CONCLUSION. 


6. An RDMS meeting the all the criteria providing benefits to the Alaska 2000+ Range 
Improvements Project does not exist, not yet, anyway. However, demonstrations 
conducted by Litton Data Systems, Sensis Corp., and GE Syracuse, have clearly shown 
that industry possesses the capability to meet our established criteria. A RDMS at the 
LRR would integrate radar data from both the AN/FPS-117 and AN/TPS-63 radars into 
the ROCC, 3DS, and ACMI while at the same time increasing the efficiency of the entire 
Alaska 2000+ Range network. 
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ACTION RECOMMENDED. 


7. Procure a device, such as a RDMS, located at the LRR, which will provide the radar 
data management capabilities as described above. 


GEORGE A. PAHLS, Colonel, USAF 
Vice Commander 


13 Atch 


1. RDMS Integration Scheme 

2. Radar Data Output Comparison Bar Graph 

3. Radar Scope Comparison Data 

4. Analytical Data: From Litton Data Systems Final Rpt 
5. Analytical Data: Sensis Corp. From Final Rpt 

6. Litton data, stating plot and track message output 
7. Sensis data, stating correct message formats 

8. Sensis data, integration capability with CTIS 

9. Sensis data, integration capability with TPS-63 

10. Litton Data Systems Itr, integration with TPS-63 
11. Sensis Corp. stating open architecture design 

12. Sensis Corp. statements on workstations rec/play 
13. Litton statement on workstations, record/playback 
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Figure l-la. Accumulation of all Target Reports from Kotzebue 
Radar, June 1991 - Large Concentration of Clutter Reports from Ice 
Floes and Other Sources Makes Manual Track Initiation Difficult. 
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Figure 1-1b. Accumulation of all Target Reports from Kotzebue Radar 
as Output by ATS, June 1991 - Most Clutter Reports from Ice Floes 
and Other Sources Were Removed, Making Manual Track Initiation Less 
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Figure 1-2b. Number of all Kotzebue Radar Target Reports Output Each 
Radar Scan by ATS, June 1991 - The Number of Target Reports for Which 
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Figure 1-3a. Accumulation of all Radar Search Target Reports from 
Kotzebue Radar, June 1991 - Removal of IFF Target Reports Gives a 
Representation of Clutter 
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Figure 1-3b. Accumulation of all Radar Search Target Reports from 
Kotzebue Radar Output by ATS, June 1991 - There Were Few True 
Targets without IFF in the Surveillance Region 
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Figure 1-4a. Number of Kotzebue Radar Search Reports Output Each 
Radar Scan, June 1991 - Removal of IFF Targets Reduces the Number of 
Targets from 103 to 8 
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Figure 1-4b. Number of Kotzebue Radar Search Reports Output Each 
Radar Scan by ATS, June 1991 - Three of These Reports Are Due to Air 
Targets, Others Are Due to Clutter 
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mecure }-/a. AIS Track of F-15 Air Interceptor (Intruder), Tatalina 
Radar Site, October 1991 - The ATS Maintained Track Even During 
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Figure 1l-/7b. ATS Track of F-15 Air Interceptor (Defender), Tatalina 
Radar Site, October 1991 - The ATS Maintained Track for all Except 
One Radar Scan Even During Rapid 180 Degree Heading Changes 
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3, DEMONSTRATI DATA COLLECTION AND ANALYSIS 


3,1 False Alarm Rejection Performance 


Captain Pierce supervised the connection of the MSC and the collection of data from 
four AN/FPS-117 sites: Tatalina, Tin City, Kotzebue, and Cape Romanzoff. Radar 
output data was collected from each site for a period of approximately 20 minutes 
for the purpose of determining the false alarm rejection capability of the MSC. The 
collected data was analyzed off-line using Data Analysis and Display Software 
(DADS) previously developed by Sensis. A summary of the false alarm rejection 
results is presented in Tables 3.1. through 3.3 


Table 3.1 False Alarm Rejection of the MSC on FPS-117 Radars 


Primary Radar Reports 


False Alarm 
Radar Site MSC Input MSC Output Rejection Ratio 


Kotzebue 2) Stoel 
Tin City 30.0 to 1 
Cape Romanzoff 10.5 to 1 
Tatalina 12.6 to 1 





Table 3.2 False Alarm Rejection of the MSC ona 2D Radar 












Primary Radar Reports 


False Alarm 
Radar Site MSC Input MSC Output Rejection Ratio 


Table 3.3. Comparison of the MDP AND MSC 


Primary Radar Reports 


False Alarm 
Radar Site MDP Output MSC Output Rejection Ratio 





The DADS program was also configured to display all scans of the recorded data 
(approximately 100) for each site. Once the data file playback had been completed, 
the screen dump feature of the DADS program was used. This enabled the user to 
"take a snapshot" of the display and transfer the output to a laser printer where a hard 
copy is printed. The diamond-shaped symbols ‘with a plus sign in the center 
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AN/GYQ-51 Advanced Tracking System (ATS) 


A High Performance Extractor/Tracker 
for Improved Air Surveillance 


INTRODUCTION 


The AN/GYQ-51 Advanced Tracking System (ATS) is a post processor with radar 
beacon target extractor and tracker capabilities to significantly improve air 
surveillance effectiveness. The ATS has many advanced design features that 
distinguish it from conventional extractor equipments to provide improved small 
target detection in clutter. Its embedded adaptive tracker automatically 
creates and maintains a target track file which facilitates interoperability 


with systems requiring filtered plot or track data inputs. These include fixed 
site Operations Centers (SOCCs/ROCCs) and mobile systems like the AN/TYQ-23 


Modular Control Equipment (MCE). ATS has been rigorously evaluated and tested 
by the USAF and selected exclusively as the discriminator/tracker for the TAC 
AN/TPS-43E and AN/TPS-75 radars used with the USAF MCE, and with the USAF 
aerostat radar program. It is also presently employed in the U.S. Customs 
Service Southwest Tethered Aerostat (STAS) Program. Requirements have been 
established for 56 additional USAF units, 18 USCS units, 1 ROKAF unit, with 
emerging requirements for 28-30 more ATSs. As such, the AN/GYQ-51, Advanced 
Tracking System, has become the USAF and USCS standard post processor/advanced 
tracker. 


Recent live ATS Demonstrations in Alaska (with the General Electric AN/FPS-117) 
and High Rock, Bahamas (with the Westinghouse TPS-63) prove its adaptability to 
interface with a wide variety of older and new-generation radar, systems. 


IMPROVED DETECTION AND FALSE ALARM PERFORMANCE 


The ATS upgrades air surveillance system performance to improve overall mission 
effectiveness. The ATS improves radar detection performance by allowing radar 
detection processing to operate at maximum sensitivity and applying post 
detection integration in the tracker to extract true targets from the 
background plot noise. The radar and target extractor within the ATS operate 
to achieve optimum small target (low RCS) detection by accepting all the 
radar’s plots, then allowing the tracker to perform track-before-detect 
processing by applying multi-scan integration. This process uses all available 
plot data to determine the presence of true targets from amid clutter in the 
surveillance volume. 


Employing this two stage detection process allows the ATS to achieve optimum 
detection performance without sacrificing false alarm performance. 
Conventional extractors do not employ post detection integration and 
consequently compromise small target detection to preserve an acceptable false 
alarm rate. Most of that conventional false alarm reduction is accomplished 
through "fixes" to the radar (e.g. sector blanking, raising the elevation beam 
angle, raising the sensitivity level). These fixes do reduce clutter in the 
affected area, but at the price of not detecting all of the real aircraft. The 
ATS achieves its improved detection performance without these radar fixes, 
through adaptive target extraction, noisy area tracking, and automatic target 
discrimination. 


107 


In applications of the ATS with the FPS-117, the radar continues to perform all 
the target extraction functions. The ATS receives the radar’s plot data via 
the message output channels. ‘The FPS-117 does not produce the conventional 
radar video that would allow the ATS to perform the plot extraction process (as 
with the TPS-63). Instead the ATS extractor is by-passed and the plot data 
from the radar is processed by the tracker in the very same way it processes 
internally generated plot data. However, the FPS-117 plot extraction function 
should be optimized for maximum sensitivity and probability of detection (PD), 
with none of the frequent fixes which reduce false alarms, but also reduce the 
opportunity to detect real aircraft as well. 


Noisy Area Tracking 


The ATS advanced tracker uses adaptive noisy area tracking techniques and 
automatic target discrimination to determine the presence of true targets and 
reject false alarms. The objective is to preserve the optimum PD achieved in 
the extractor, and eliminate false alarms to an acceptably low level at the 
output of the ATS system. Plot data from the extractor function is 
automatically processed by the high capacity, high performance adaptive 
tracking function within the ATS. Noisy area track initiation is performed on 
all input data. The target behavior model defines the envelope of target 
dynamics that meet the true target criteria used by the multi-scan track 
initiation process. It is through this process that the true targets are 
discriminated from the background false plots. 


The track initiation function is capable of rejecting large numbers of false 
plots in the process of testing all possible plot combinations for true target 
characteristics. Typically, many thousands of trial tracks are created and 
tested in performing this operation. Most are rejected as combinations of 
false plots that exhibited brief periods of true target behavior. With few 
exceptions, only true targets in the surveillance volume become established ATS 
tracks. Once those tracks are established for the true targets, the 
established tracks are used as the target true reference to automatically 
discriminate plots from the radar extractor (i.e., sort out the noise and 
clutter). The resulting output consists of the true targets and very few false 
reports. 


Automatic Target Discrimination 


Based on the measured false plot density within individual surveillance 
regions, the ATS determines the eligibility of plot for output to the 
Operations Center. In clear areas (receiver noise only), plots are output 
without discrimination, as the likelihood of plots being true targets in these 
clutter free regions is high. In those areas where the false plot density 
indicates a noisy region, plots must correlate with established tracks to be 
output by the ATS. Beacon plots and their reinforcing radar returns are always 
output without discrimination. This plot filter criteria is therefore an 
intelligent selection process based on measured noise levels and established 
target tracks. This differs significantly from the conventional, totally 
censored clutter areas where target detection goes to zero. This capability 
has been demonstrated as crucial in meeting PD and false alarm performance 
requirements in the latest generation radar applications. Detection and false 
alarm rate requirements are such that post detection processing is the only 
workable solution for eliminating the false plots created by highly sensitive 
radar systems (e.g. FPS-117s and Aerostats). 
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INTEROPERABILITY 


The ATS has proven essential in providing a high PD and low false alarm plot 
input to Operations Centers for a variety of radars. The SOCCs/ROCCs for 
example, are not tolerant of false alarms, as is true of many Similar centers. 
Their central processors can handle no more than 10 to 15 false alarm plots per 
scan per radar, 1n order to successfully process all of the system’s radar 
data. Otherwise, communications links from the remote radars to the SOCC/ROCC 
become overloaded, and/or the Q-93 computer system’s processing capacity 
becomes stressed and slowed, creating significant operational degradation, or 
prompting implementation of those radar fixes which "unplug" or "sector-out" 
entire sections of the regional surveillance volume. The ATS’s capabilities to 
reduce clutter at the radar site, while increasing PD over true targets, has 
been field-proven to alleviate many of these current systemic SOCC/ROCC 
problems. 


CONCIUS ION 


The ATS is a proven, high performance post-processor/advanced tracker that 
improves surveillance performance to meet small, low flying target PD 
requirements. By the use of automatic and adaptive extraction and tracking, 
the ATS enhances radar performance for small target detection and false plot 
control. The ATS is particularly effective in high clutter enviromments caused 
by weather, birds, sea state, topography and even ECM. The ATS also provides a 
local target track file at the radar to support automatic track data tell to 
Operations Centers, offering substantial improvement in maintaining continuity 
of operations with a distributed target data base. 


The ATS is now accepted as a fielded necessity for detection and tracking in 
many early warning/air surveillance environments, achieving here-to-fore 
unobtainable automation of the surveillance function. 


February 1992 
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Captured "pictures" of the ASD and printed hard copies on a laser jet printer. 


Successfully decoded the "ROCC Status" message in real-time and displayed 
the results in man-readable textual format on the ASD. 


Successfully generated real-time measures of radar performance such as: 
message counts per scan; parity errors (per channel); bad or unrecognized 
messages; beacon reinforcement rate; primary reinforcement rate; and track 
quality. 


Successfully recorded radar data for two sites simultaneously and played the 
data back for post-mission analysis in either Plan Position Indicator (PPI) or 
Range-Height Indicator (RHI) format. 


Provided numerous ASD features including zoom, offset, reorigin, track 
blocks, track vectors, tabular displays, SIF locate, and continuously updated 
range and bearing calculations. 


Using Sensis developed "Data Analysis and Display Software" (DADS), 
successfully displayed and analyzed recorded radar data. 


Rejected false alarms far better than the MDP algorithm implemented within 
the DDP -- the MSC further reduced the false alarm rate out of the MDP on the 
AN/FPS-117 at Tin City by a factor of 39.6 to 1. | 


The important operational capabilities which Sensis can provide in a singularly cost- 
effective way include: 


e 


Enhancing range-safety throughout the Yukon MOA by optimizing the 
performance of the AN/FPS-117s and the AN/TPS-63 via a two step process: 
one, increasing radar sensitivity at low elevations and through valleys by 
proper selection of detection thresholds, STC attenuation and elevation beam 
pointing angles; and two, employing MSCs to effectively control false alarms. 


Integrating ROCC track information into the CTIS network by developing a 
special interface board (a derivative of the UNIO™ board within the MSC) 
which taps into the data stream broadcast from the Display Controller to the 
HMD-22(J) console. 


Providing radar plot and track data on an Ethernet LAN where it can be picked 
off and processed by new “open architecture" data processors and displays 
without interfering with current ROCC operation. The key to this realizing this 
capability is to put MSCs at the radar sites to reduce the amount of plot data 
(by eliminating false alarms) to the point that the track data also available'from 


the MSCs can be added and transmitted to the ROCC over existing data links. 


By presenting data and supportive discussion, the following report sections 
substantiate the assertions made here relative to the level of demonstrated 
performance and the potential for improved operational capability. Estimates of the 
cost and schedule of several specific programs proposed to enhance the ROCC's 
operational capability are also included. 
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¢ Captured "pictures™ of the ASD and printed hard copies on a laser jet printer. 


- Successfully decoded the "ROCC Status" message in real-time and displayed 
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capability is to put MSCs at the radar sites to reduce the amount of plot data 
(by eliminating false alarms) to the point that the track data also available'from 
the MSCs can be added and transmitted to the ROCC over existing data links. 
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performance and the potential for improved operational capability. Estimatcs of the 
cost and schedule of several specific programs proposed to enhance the ROCC's 
operational capability are also included. 
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4.2 AN/TPS-63 Improvement 


The key to improving the performance of the AN-TPS-63 is simply to reduce the 
number of false alarms it generates through the use of the MSC. Our conservative 
estimate is that the MSC will reduce the AN/TPS-63's false alarm rate by more than 
10 to 1. This judgment reflects the performance which the MSC achieved on the 
ARSR-3 at Kenai (i.e., a 28 to 1 false alarm rejection ratio) and our experience with 
an aerostat-borne AN/TPS-63 in Yuma Arizona. The MSC cut the false alarm rate 
out of the Yuma radar by 15 to 1, and because the moving aerostat has line-of-sight 
to a great deal of vehicular traffic, the problem is far more difficult. An example of 
the performance witnessed on the Yuma AN/TPS-63 is provided in Figures 4.8 and 
4.9. They indicate the multi-scan search-only outputs at the input and output of the 
MSC, respectively. 


Two unique interfaces are available to handle AN/TPS-G63 plot outputs. The USMC 


radar utilizes an NTDS parallel interface and other versions utilize a three channel 
serial output format. Sensis has experience with both interfaces. The MSC has the 
demonstrated capability to accept NTDS format messages, perform false alarm 
rejection processing and output the data over ‘two serial, synchronous data channels 
in the ROCC standard message format. It also has the demonstrated capability to 
accept data from the aerostat borne radar over three serial channels, perform false 
alarm rejection processing and output the data (with no data loss) over two serial, 
synchronous data channels in the ROCC standard format. 





By 


Litton 


BONO Woolley Ave tite: 


Data systems Cece tars 


HAO 7001 


HI OL TOO 


10 February 1992 


Captain Mark Pierce 

llth Tactical Control Wing/DOx 
6900 9th Street, Suite #301 
Elmendorf AFB, AK 99506 


Dear Captain Pierce: 


Enclosed find the requested Advanced Tracking System (ATS) specification 
information. It contains detailed ATS specification data on applicable 
government documents/Mil-Standards, system and functional characteristics, 
logistics and reliability, and basic Quality Assurance/Verification Test 
provisions. 


The ATS is a military nomenclatured system, the AN/GYQ-51, which has been in 
successful field use by the Air Force, U.S. Customs Service and Air National 
Guard for several years. In addition, new ATS systems have recently been 
delivered to Raytheon Corporation for use in the Warner Robins AFB PAVE PAWS 
program, to the Korean Navy Tactical Data System (KNIDS) and to General 
Electric for the HYPAR program in Turkey. Fourteen more systems are in 
production at this time for a variety of other customers, including NORAD. 


The continuing success and reliability of ATS stems from Litton’s extensive 
history of providing state-of-the-art, automatic target detection and tracking 
equipments to interface with various search radars and IFF sets. Between 
1979-85, Radar Beacon Digitizers (RBDs) were delivered for diverse military and 
Civil applications in Iceland, Ft. Huachuca, several Navy FACFACs, Berlin’s 
Tempelhof airport, and the USMC’s Marine Air Traffic Control and Landing System 
(MATCALS). Litton’s newest tracker, the ATS, began development in 1981, and 
led to the MCE Interface Group (MIG) which performs stand-alone target 
extraction and advanced tracking for the Air Force’s MCE system. The latest 
ATS version, commonly referred to as "STAS" (for the Southwest Tethered 
Aerostat program), is the most highly refined automatic tracker available. 


It 1S important to note that, for potential Alaskan applications, the ATS 
interfaces with a variety of radars other than the FPS-117. That includes full 
interoperability with the TPS-63 or ARSR-3 series radars, not to mention 
tactical radars that may deploy to the theater. 








Please contact me with queries, or any requests for follow-on data. 


AtAdp 


Jack Griff 
Program Manager 
Air Operations 
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2. DEMONSTRATION CONFIGURATION 


A block diagram of the MSC hardware configuration employed in the demonstration 
is shown in Figure 2.1. As shown therein, two MSCs were used simultancously. 


Ethornot LAN 
a 
2 Channels Muitl-Scan 2 Channols 
per Radar Correlator per Radar 
#1 eo Deta Output 
to ROCC 
Digital Switch 








2 Channels Multl-Scan Lae 2 Channels 
per Radar gee Corelstor per Rader 
#2 
Display & | 
Control 


Pictne Zal Demonstration Hardware Configuration 


Both units were placed on top of the Digital Switch located in the ROCC computer 
room. The MSCs were connected to serial maintenance ports of the digital switch 
through standard DB25 cables. Data processed and filtered by the MSCs was 
transmitted onto an Ethernet Local Arca Network (LAN) using TCP/IP protocol. 
The cable was laid under the floor to a SUN SPARCstation display and control 
terminal placed next to a ROCC Radar Display Unit (RDU). 


Within 30 minutes after being unpacked, and with no hardware adjustments or 
modifications, the MSC equipment was properly set up and processing live radar 
data. The setup was accomplished without interrupting or, in any way, interfering 
with system operation. Jumper wires were used to connect the radar data and clock 
to the appropnate input and output pins on the MSC. 


2,1 Multi-Scan Correlator 


The MSC is a commercially available, in-production processor which reduces clutter 
induced false target reports without materially reducing target detectability. In fact, 
the significant reduction in false alarms realized by the MSC peimits an incrcase in 
radar sensitivily, which can actually increase target detectability. The hardware 
consists of COTS sub-assemblies with the exception of one digital board which was 
specifically designed to efficiently handle a varicty of serial synchronous and 
asynchronous message formats. It utilizes industry standards such as the VERTX 
real-time operating system, a VME bus interface, an Etheret LAN, a RETMA 
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chassis and a TAF standard SUN workstation. Display and processing code is 
written tin "C", a High Order Lunguage (HOL). 


The MSC is placed between the radar and communications modem, and performs the 
correlation processing required to eliminate reports from reflectors which don't move 
like true air targets. Returns which are eliminated this way include surface and 
volumetric clutter, long-range clutter induced by anomalous propagation (AP), and 
slow-moving targets such as vehicular traffic. birds and shipping vessels, Beacon 
and search-reinforced beacon returns are tracked internally but not removed from the 
data stream. The MSC is capable of handling up to 1600 reports per 12 second scan 
(twice the current data link capacity). A more detailed description of the hardware 
and software of the MSC is found in Appendix A. 

Physically, the MSC 1s housed in a standard’19" RETMA rack mount chassis, 5 
inches high and 24 inches deep. The chassis contains an internal power supply, two 


fans, an MVYME147-1 68030 processor board and a serial interface board. The 
weight is less than 20 pounds and the power consumption ts less than 150 Watts. 


2,2 Ethernet LAN 


All control communications and data transfer between the MSCs and the SUN 
workstation 1s accomplished via the Ethemet LAN. The modular nature of the 
system interface provides the capability to network several units together to obtain as 
many input ports as necessary. The bandwidth of the Ethernet LAN, 1 Megabit per 
second, is sufficient to accept radar inputs from over 100 sources at 4800 PPS. All 
of the incoming radar data 1s broadcast on the Ethernet LAN where it is available to 
be distributed to other non-proprietary computer systems which may be used in the 
future. 


2 PAR ion 


The SUN SPARCstation is a high performance processor with a high resolution 
display and built-in mass storage capability. The SUN utilizes the UNIX operating 
system and windowing to provide an efficient and convenient method for defining 
and generating displays and controls. 


The Display and Control system implementation makes use of a keyboard and mouse 
or trackball for operator entry and selection of display information. Radar data and 
Operator entrics can be recorded and stored on the 300 Megabyte hard disk and 
transferred to magnetic tape for data archiving. 


2, DEMONSTRATION CONFIGURATION 


A block diagrain of the MSC hardware configuration employed in the demonstrauon 
is shown in Figure 2.1. As shown thercin, two MSCs were used simultancously. 
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igure 24) Demonstration Hardware Configuration 


Both units were placed on top of the Digital Switch located in the ROCC computer 
room. The MSCs were connected to serial maintenance ports of the digital switch 
through standard DB25 cables. Data processed and filtered by the MSCs was 
transmitted onto an Ethernet Local Area Network (LAN) using TCP/IP protocol. 
The cable was laid under the floor to a SUN SPARCstation display and control 
terminal placed next to a ROCC Radar Display Unit (RDU). 


Within 30 minutes after being unpacked, and with no hardware adjustments or 
modifications, the MSC equipment was properly set up and processing live radar 
data. The sctup was accomplished without interrupting or, in any way, interfering 
with system opcratiou. Jumper wires were used to connect the radar data and clock 
to the appropriate input and output pins on the MSC. 


2,1 Multi-Scan rrelator 


The MSC 1s a commercially available, in-production processor which reduces clutter 
induced false target reports without materially reducing target detectability. In fact, 
the significant reduction in false alarms realized by the MSC permits an increase in 
radar sensitivity, which can actually increase target detectability. The hardware 
consists of COTS sub-assemblics with the exception of one digital board which was 
specifically designed to efficiently handle a variety of serial synchronous and 
asynchronous message formats. It utilizes industry standards such as the VERTX 
real-time operating system, a VME bus interface, an Ethernct LAN, a RETMA 
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chassis and a TAF standard SUN workstation. Display and processing code ts 
written in "C", a Hrgh Order Language (HOL). 


The MSC is placed between the radar and communications modem, and performs the 
correlation processing required to eliminate reports from reflectors which don’t move 
like true air targets. Returns which are eliminated this way include surface and 
volumetric clutter, long-range clutter induced by anomalous propagation (AP), and 
slow-moving targets such as vehicular traffic, birds and shipping vessels, Beacon 
and search-reinforced beacon returns are tracked internally but not removed from the 
data stream. The MSC is capable of handling up to 1600 reports per 12 second scan 
(twice the current data link capacity). A more detailed description of the hardware 
and software of the MSC is found in Appendix A. 

Physically, the MSC is housed in a standard:19" RETMA rack mount chassis, 5 
inches high and 24 inches deep. The chassis contains an internal power supply, two 


fans, an MVME147-1 68030 processor board and a serial interface board. The 
weight is less than 20 pounds and the power consumputon is less than 150 Watts. 


2,2 Ethernet LAN 


All control communications and data transfer between the MSCs and the SUN 
workstation is accomplished via the Ethernet LAN. The modular nature of the 
system interface provides the capability to network several units together to obtain as 
many input ports as necessary. The bandwidth of the Ethemet LAN, 1 Megabit per 
second, is sufficient to accept radar inputs from over 100 sources at 4800 PPS. All 
of the incoming radar data is broadcast on the Ethernet LAN where it is available to 
be distnbuted to other non-proprietary computer systems which may be used in the 
future. 


Zz PARCstatior 


The SUN SPARCstation is a high performance processor with a high resolution 
display and built-in mass storage capability. The SUN utilizes the UNIX operating 
system and windowing to provide an efficient and convenient method for defining 
and gencrating displays and controls. 


The Display and Control system implementation makes use of a keyboard and mouse 
or trackball for operator entry and selection of display information. Radar data and 
Operator entrics can be recorded and stored on the 300 Megabyte hard disk and 
transferred to magnetic tape for data archiving. 
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4 Recording and Playback bili 


The Display and Control station will permit the user to record the received data on 
the SUN SPARC workstations internal hard disk memory unit and playback the data 
under operator control, in an operator designated file, assuming recording is enabled. 


The SUN workstation is equipped with a 300 megabyte internal hard disk drive with 
a 150 Megabyte back-up magnetic tape drive for recording and archiving data. The 
Display Station can be utilized to record data from. multiple radars simultaneously 
for later replay and analysis. Additional display stations can be added as 


requirements arise and one of the Display Stations could subsume the functions of 
the present RDU with a full color high resolution display. 


The playback of previously reccrded radar messages is also contivlled via the 
Display and Control Station. The operator has the capability to specify the playback 
filename and to control the rate of playback. The rate of playback can be varied 
from a real-time rate to approximately 10 times the real-time rate. This feature is 


useful for viewing data files to search for specific charactenstics or events. In 
addition, the file playback may be paused and restarted under operator control. 


9.5 Systein Features Suriunary 


In summary, the MSC offers the following features to the user: 


¢ Modular Architecture: supports single-site remote installations or networking 
of multiple MSCs at the C3I Cen.er 


* Programmable "Universal" Interface Processor: provides transparent 
connection with existing data link protocols and the capability to perform 
channel data conversions through re-bauding and/or channel muxing 


¢ Non-Volatile Storage of Site-Specific Parameters: for automatic self-activation 
upon power-up 7 

¢ On-Line Control: permits user selection of certain parameters such as active 
and bypass regions, velocity gates and clutter maps 


« Embedded Ethernet Local Area Network (LAN) Port: provides a common 
communication path for growth options including the CMC-2 subsystem and 
the networking of multiple MSCs 


¢ Equipment Status: via built-in-test with front-panel indicators and status report 
messages to the CMC-2 


¢ On-Line Performance Monitoring: performs real-time performance monitoring 
and health asscssment with fault logging and automatic bypass in the event of 
a detected fault 
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Tracker Data Analysis System (TDAS) 


The TDAS is a PC-based program for recording and analyzing ATS output data. 
Provisions are made for recording of (1) the raw plot extracted data, (2) the 
filtered plot reports generated by the ATS, and (3) the ATS track file as 


output to users who require Established tracks rather than filtered plots. 


TDAS is also used for analyzing recorded data’ and presenting the analyzed data 


for display. Provisions are made for hard copy printout of displayed data. 





In the Alaska ATS Demo, TDAS was used for generation of all the data 


presentations for this report. 


A few of the TDAS capabilities pertinent to the ATS demonstration, and 


subsequent data analysis, are listed here: 


1. Generate a geopositional display of radar target reports, filtered 
target reports or tracks. This display has most of the capabilities of 
the normal radar PPI display including radar centered display, display 
offset to cursor (ball tab) position, and range scale variation. 
Provisions for data retention for any number of radar scans is provided. 
Provisions are also made for stepping through recorded data one radar 
scan at a time (with data retention for as many radar scans as the 


operator chooses), clearing of the display, and continuation of the 


process. Hard copy printouts can be made at any step in the process. 


2. Generation of scan sequence diagrams which present data values for each 
individual radar scan (up to 250 radar scans on a single display). Up 
to four scan-sequence diagrams can be generated at one time. Values of 
any track or plot parameter output by the ATS can be displayed. 

Examples are: 

a. The number of target reports output by the Target Detector each radar 
scan. 

b. The number of plot reports output by the ATS each radar scan. 
The number of tracks output by the ATS each radar scan. 


d. Speed, heading, height and track quality for any ATS output track. 


3. Generation of histograms of data maintained in the ATS track file and 


oueput Dy the ATS plot filter. For example, the distribution of tracks 


with speeds in 25 mph speed intervals for speeds from zero to 500 mph. 
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APPENDIX D 


EXPERIMENT PLAN AND REPORT FORMATS 
GENERAL: Ideally, an experiment is reported on in two documents, an 
experimental plan that lays the foundation, and an experimental report that tells 
what actually took place and what the results were. For CC4003 we do not 
have time to produce elaborate experimental plans, so we move some of the 
information into the report. The following outline is suggested. Parenthetical 
notes identify information that could be in the plan, the report, or both. 
1. Introduction (plan -- summarize in report) 
A. Introduce the team 
B. Purpose 
(1) The real world problem the experiment will help solve 
(2) The specific questions the experiment seeks to answer 
(3) The approach 
(4) Anticipated Results 
C, Scope of Experiment 
2. Experimental Design (plan --summarize in report) 
A. Setup 
(1) Physical 
(2) Test subjects 
(3) Special Equipment 
(4) Schedule of Trials 
B. Hypotheses 
C. Assumptions 


D. Statistical Design of Experiment 
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E. Measures 
F. Instrumentation 
G. Testing & Pilot Trials 
3. Data Description (both: plan -- what will be collected, assumptions; 
report -- details) 
A. Example of raw data 
B. Data problems 
C. Data coding scheme 
D. Data table 
E. Data Reduction 
4. Analysis (both: plan -- how it will be performed; report -- details) 
A. Analysis Plan (plan -- summarize in report) 
B. Methodology (report) 
C. Results of Analysis (report) 
D. Any additional assumptions that were required (report) 
5. Conclusions (report) 
A. Hypotheses Results -- Interpretations 
B. Other Interpretations 
C. Real world meaning of results 
D. Experiment Summary 
6. Recommendations (report) 
A. Changes to the experiment 


B. Continuation of the Experiment 
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APPENDIX E 


EVALUATION OF STUDIES PARADIGM 


Evaluating Studies 
CC4003--January 1994 


OBJECTIVE clear & relevant | easy to understand, | 
unambiguous, appropriate 

ASSUMPTIONS | reasonable | logical, realistic, sensible, | 
minimum 

HYPOTHESES germane testing them accomplishes 
objective 










EXPERIMENTAL] comprehensive 


pertinent, observable, 





DESIGN but executable 


practical 





MEASURES quantifiable numerical, accurate, 
and discriminatory significant, important 
DATA replicable | precise, well-defined, 
reliable 
ANALYSIS correct and careful, scientific, 
unbiased Statistical, reality 
CONCLUSIONS | supportable appropriate, valid, logical 
Consistent compatibility, direct 
linkage, pertinent 
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DEFINITIONS OF CHARACTERISTICS-EVALUATION PARADIGM 


The objective is clear and relevant if it is easy to understand, unambiguous 
and appropriate to the goal of the study. 


The study is simple if it is short and illustrative, has a minimum number of 
assumptions, alternatives, hypotheses and measures plus easily understood 
analysis and conclusions. 


The assumptions are reasonable if they can be supported logically, 


realistically and sensibly. They should be the minimum necessary to limit the 
scope of the problem to viable choices. 


The experimental design is comprehensive but executable if it tests all the 
pertinent hypotheses with measurable data by a practical analysis plan. 


A hypothesis is germane if its test efficiently accomplishes the objective of 
the study. 


Measures are quantifiable if they provide numerical values that accurately 
reflect the alternatives. The are discriminating if significant differences in their 
values reflect important differences in the alternatives. 


Data is replicable if it is precisely defined and accurately measured and 
processed. A test of replicability is if repeated trials would produce the same data. 


An analysis is correct if it carefully tests data with Statistically sound methods. 
It is unbiased if the assumptions and methods applied reflect reality. 





The conclusions are supportable if they are valid results of an analysis that 
follows logically from the study. 


The study is consistent if there is compatibility in scope and detail at each 
step. In addition, direct linkage between the steps must maintain continuity with 
the objective. 
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APPENDIX F 


REVISED EXPERIMENTAL PLAN 
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I. REVISED INTRODUCTION 


A. PROJECT MANAGEMENT 

The 11 ACW is going to conduct an experiment to assess the capabilities of 
the Litton Data Systems Advanced Tracking System (ATS). 

1. In order to manage the demonstration, an Advanced Tracking System 
Project Management Team was established. The demonstration team leader is 
11th ACW/DOX; other team members are listed in paragraph three. 

2. The following agencies are participating or providing support for the 
ATS demonstration: 


a. 11 ACW/DOXXQ_ -- Capt Pierce 


b. 11 ACW/DOP -- SSet Myher 

c. 11 ACW/LGOR -- MSet DeLuca 
d. 11 ACW/LGKC -- MSet Shuler 
e. 11 ACW/LGKM -- TSgt Cushman 
f. 744 ADS/DOO -- Lt McNeil 


ga 


21StTFW/DOW _ -- Capt Hill 


B. PURPOSE 

The purpose of this experiment will be to determine whether or not Litton 
Data Systems Advanced Tracking System (ATS) is capable of more efficiently 
managing the flow of radar data coming into the Region Operations Control 
Center (ROCC). Managing the radar data more efficiently will enable the 
operators to realize the full operational capabilities of the new radars without 
overloading the ROCC Central Computer. As a result, the demonstration 1s 
designed to provide insight into the radar data management problems being 


experienced by the Alaska NORAD Region (ANR) and the ability of the ATS to 
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solve them. Specifically, the ANR would like to determine the validity of the 
claims of the ATS's capabilities. For example: Can the ATS accurately 
discriminate true targets from clutter? From an air defense point of view, can the 
ATS provide target discrimination without target elimination? 

The demonstration will select three sites, all equipped with ECPs. All of 
the ECPs will be tumed off for the demonstration in order to reestablish the radars 
to a fully sensitized state. The ATS must prove that it is capable of handling the 
data workload from a fully sensitized radar. 

The ATS will not be installed at the selected radars because of the logistics 
and prohibitive costs. Instead, the ATS equipment will be installed in the ROCC. 

The ATS will be connected at the ROCC Communications Segment, the 
point where the data arrives in the ROCC via satellite from the selected radars. 
The data arriving at this point has not yet been processed by the ROCC central 
computer. Therefore, connecting the ATS at the ROCC Communications 
Segment simulates the ATS processing the data at the radar site. 

The next step will be to compare the amount of data a fully sensitized radar 
forwards with the amount of data a fully sensitized radar equipped with an ATS 
forwards. The ROCC provides the capability to simultaneously measure ATS and 
non-ATS data from the same site (see Figure 4)[Ref. 10]. The reduction in data, 
and the quality of the data must be analyzed simultaneously. The ATS may live 
up to its clutter reduction claims, but the data has to be analyzed to ensure that 
the ATS is not eliminating or erroneously manipulating valid air targets necessary 
for air defense and air training missions. In order to verify the target data, a 
specific number of aircraft, flying different profiles, will be used. The experiment 
will be designed to control the number of aircraft. However, the actual flight 


profiles along with the number of aircraft used during the test will only be known 
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by a third party, disassociated from the experiment. Only after the experiment is 
completed and the analysis turned in will the aircraft portion of the experiment be 
revealed. The purpose of this single blind design is to eliminate any bias from the 
experimenters. 

The anticipated results: Measurable differences between the quantity of 
data received from a radar in a fully sensitized state, when equipped with an ATS 
and when not, are expected. However, whether or not the ATS eliminates valid 


air targets remains to be seen. 


C. THE SCOPE OF THE DEMONSTRATION: 

This demonstration will provide the information the 11th AF Commander 
needs to decide whether or not the ATS (or other associated Radar Data 
Management System) provides viable solutions to the radar data management 
problems experienced by the ANR. There are several potential benefits if the ATS 
demonstration proves successful. First, ECPs can be eliminated from the radars, 
restoring them to their full operational capability, and providing greater low level 
detection of aircraft and missiles. In the area of training, the improved detection 
capability will increase the radar coverage in the low level training areas. 

Benefits should also be seen in the ROCC central computer. Installing the 
ATS at the radar head will decentralize the computer processing. The ATS will 
discriminate between clutter and target data, dump the clutter data, and pass only 
the target data to the central computer for further processing. The main benefits 
to the central computer are reductions in the amount of processing required and 
the potential for a computer overload that arbitrarily dumps data. 

The reduced computer workload will enable the installation of additional 
radars. This is important because PACAF wants to expand the air training ranges 


in Alaska. The eruption of Mount Pinatubo and subsequent loss of a major air 
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training exercise in the Philippines called Cope Thunder required more training, 
and thus an expansion of the air training ranges, in Alaska. As a result, additional 
radars are needed to provide coverage in these new training areas. However, the 
additional data from these sites could easily create an overload condition in the 
computer. Dumping data during a major exercise would seriously jeopardize the 
Safety of the entire mission. 

The life of the central computer will be prolonged because the amount of data 
the central computer has to process will be eased. Thus, the costly acquisition of 


a new generation ROCC computer can be delayed. 
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Il. REVISED EXPERIMENTAL DESIGN 


A. SETUP: 
1.‘ Physical 

The physical setup is the crux of the entire experiment. The 
demonstration would not be possible if not for the flexibility of the ROCC 
system. Figure 4 depicts the actual physical interface between the ROCC and the 
ATS equipment. The following is a discussion of the ROCC-ATS interface. 

The digital switch provides the interconnection between the 
communication channels from the radar sites and the central computer. The radar 
data at the digital switch has not been processed by the ROCC computer. Each 
radar sends its data to the digital switch at the ROCC via three 2.4 kbps 
communications channels. These channels feed directly into three ports 
associated with each radar at the digital switch. Two ports, designated A and B, 
are normally interconnected with the primary and standby central computers. 
The third port, designated C, is available for recording data on a noninterference 
basis. This is the location of the ROCC-ATS interface. At the same time, the cable 
from port B of the Digital Switch to the central computer will be disconnected at 
the computer, and the output from the ATS will be connected to the standby 
computer at its input channel (Figure 4). 

2. Factors: 

There are three experimental factors: radar sites at three levels; Litton 
ATS at two levels; and analysis equipment at two levels. Three radar sites have 
been chosen: Kotzebue, Cape Romanzof, and Tatalina (Figure 2). The ATS radar 
processing hardware/software is either on or off. Finally, the equipment being 


used to analyze the data is either third party, Air Force approved, test and 
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evaluation equipment or the Litton Data Systems analysis program associated 
with the ATS. 
3. Special Equipment: 

a. The ATS consists of all of the interface hardware as well as the 
software to conduct the data collection and analysis. All will be provided by 
Litton Data Systems. 

b. A neutral third party will be involved to provide balance and 
legitimacy to the experiment. This third party (i.e. 84th Radar Evaluation 
Squadron) must conduct the experiment in parallel with Litton. They must 
interface at the same location, with the same site, and at the same time. They will 
collect the data, analyze it, and compare their results with Litton’s experimental 
results. 

c. Aircraft will be dedicated to fly certain profiles within the coverage 
limits of the selected radar. 

4. Schedule of Trials: 

The schedule of trials will not be formally addressed in this sample plan. 
The driving factor for the schedule of trials will be the availability of the aircraft 
needed to fly the flight profiles. Making up a schedule of trials does not provide 


sufficient lessons learned. 


B. HYPOTHESES: 

la. There is no difference between the amount of data forwarded by the fully 
sensitized radar and the radar equipped with the ATS. 

lb. The alternative: There is a difference between the amount of data 


forwarded by the fully sensitized radar and the radar equipped with the ATS. 
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2a. There is no difference between the number of targets reported by the 
ROCC computer combined with the fully sensitized radar and the radar equipped 
with an ATS. 

2b. There is a difference between the number of targets reported by the 
ROCC computer combined with the fully sensitized radar and the radar equipped 
with an ATS. 

3a. There is no difference between the third party analysis results and the 
Litton Data Systems analysis results. 

3b. There is a difference between the third party analysis results and the 
Litton Data Systems analysis results. 

The first two hypotheses deal with the issues of quantity and quality of the 
data the ATS produces. Does the ATS make a difference in the amount of data 
being sent back to the ROCC central computer? And, does the ATS discriminate 
targets from clutter without eliminating valid air targets? 

The third hypothesis is designed to validate the Litton demonstration by 
determining whether or not the data is consistent with that from an established 
test and evaluation system. The 11th ACW staff must be sure that the ATS works 
as claimed and that the analysis tools that are part of the equipment package 


portray the data in an accurate manner. 


C. ASSUMPTIONS: 

1. Litton's interface hardware is compatible with the ROCC equipment 

2. The software will provide the type of analysis needed to answer the 
research questions. 

3. The weather patterns and clutter density at the three sites are 


representative of the (AOR) overall. 
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4. Both Litton and the third party Air Force evaluation equipment is capable 
of counting and keeping track of the number of targets received from the radar. 

Note: This assumption is based on the claims of Litton that its software has 
the capability to compile this type of information [Ref. 3]. Additionally, since the 
ROCC computer, a 1970s version computer, has the capability to count the 
number of tracks in the system, and 1s certified, it is assumed that the evaluation 
equipment exists within the Air Force, or else the ROCC computer could not 


have been certified in the first place. 


D. STATISTICAL DESIGN OF EXPERIMENT 

The experiment is a 3 X 2 X 2 factorial single blind experiment. This means 
that one factor has three levels and the other two factors each have two levels. 
The first factor has three levels because of the selection of three different radar 
sites. The radar sites associated with these levels are: Kotzebue, Cape Romanzof, 
and Tatalina. This factor corresponds to hypotheses one and two stated above. 
The processing equipment is the second factor. The two levels are whether or not 
the radar is utilizing the ATS. This factor also corresponds to hypotheses one and 
two. The third factor is analysis equipment. The two levels for this factor are the 
third party operationally approved equipment and the Litton analysis equipment. 
This factor corresponds to hypothesis number three. The purpose of designing a 
single blind experiment is to eliminate experimenter bias and to serve as a means 


to evaluate whether or not the ATS is eliminating valid air targets. 


E. MEASURES 

The experiment is designed to evaluate the data forwarded by the radar. The 
data will be collected and reduced into two categories. The first category deals 
with the quantity of data and the second deals with the quality of data (target 


elimination). The categories of data are listed below. 
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Category one: Quantity. The total amount of data processed and sent by 
the radar to the ROCC, measured in bits per second. 

Category two: Quality. The total number of targets being sent by the radar 
to the ROCC, measured by the total number of targets being sent by the ROCC 


computer to the operator. 


F. INSTRUMENTATION 
The instrumentation is the equipment used by Litton and the third party Air 


Force evaluation team to collect and analyze the data. 


G. TESTING & PILOT TRIALS 

Testing will be done to ensure the ATS and third party equipment can be 
connected to the ROCC digital switch. Additional testing will be done to ensure 
the commander that the experiment will not interfere with the daily operations of 
the ROCC. 

After the interfaces are made, pilot trials will be run to ensure that the data 
being collected is the correct data to test the hypotheses. The researchers will use 
the pilot testing to practice the experimental procedures. For planning purposes, 


time should be allotted in the master schedule for the pilot testing. 
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HI. REVISED DATA DESCRIPTION 


Table | presents the data coding scheme. The table depicts the three factors 
(Radar Sites, Analysis Equipment, Processing Equipment) along with their 
associated levels. The number in parenthesis is the code assigned to the particular 
level. The coding scheme will become apparent when the table listing the 


different combinations of trials is presented below. 


TABLE 1 DATA CODING SCHEME 


Radar Sites Analysis Equipment Processing Equipment 
Kotzebue (1) Third Party (1) Non ATS (1) 


Cape Romanzoff (2) Litton (2) ATS (2) 
Tatalina (3) 





The data is divided into two categories. 
Data: Total amount of data received from the radar (X in bits per second) 
Total number of tracks (Y) 

The data is divided into two categories to aid in the analysis, but it will all be 
collected at once for each trial. The computer software of the evaluation 
equipment will reduce the data into the X and Y categories designated above. 

Table 2 depicts the combinations of levels of the factors. There are twelve 
combinations of factor levels, found by multiplying one factor with three levels 
and two factors each with two levels. Each of the trials will be replicated thirty 


times in order to assume the data approximates a normal distribution. 


ios 


TABLE 2 TRIALS 


Trial # Radar Sites Analysis Equip. Processing Equip. Data (X) Data (Y) 


Oo on QA iA BW NO — 


NN Re KS NY NK KS NN KS = 
NI — NO NO NO NS NO 


I 
I 
I 
] 
Zz 
Z 
2 
yi 
3 
3 
3 
2 





There are a several steps in the data collection process. The first step is to 
collect all of the data the radar is sending to the ROCC. The total amount of data 
will be used by the analysis to answer hypotheses one and three. Target counts 
will then be compiled using the software capabilities of the computers. The target 
counts will be used to answer hypothesis number two. Additionally, the target 
count data can be used to provide supplementary information concerning 


hypotheses numbers three. 


136 


IV. REVISED ANALYSIS PLAN 


Balanced Analysis of Variance (ANOVA) will be used to analyze the data. 
The results will determine the’ outcome of the hypotheses tests. As indicated in 
the data section, there are two parts to the plan. The first section will use the 
balanced ANOVA to analyze the total amount of data. Thirty replications of each 
of the twelve trials will be made. This will be done to assume normality. The 
second phase of the analysis will also use balanced ANOVA, this time to analyze 


the thirty replications of target counts. 
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